Journal article P034

Corrections to: The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments

John Ternovski, Joshua Kalla, and P. M. Aronow (2022). The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments. Journal of Online Trust and Safety 1(2).

DOI: 10.54501/jots.v1i2.28
Audited edition: Journal of Online Trust and Safety, 2022, 1(2); 16 pages in the audited file
SHA-256 of the audited file: 698d62dd33fb7b0c1c16ffee884c581db1ca798edb0cbc0db5b77415c47b4a36
Permanent address: https://pmaronow.github.io/llm-audit/works/P034/

Summary

The paper asks whether warning voters about deepfakes makes them disbelieve real political videos. It reports two online survey experiments, one with a fictional politician and one with a real video of Mitt Romney. A warning above the video raised the belief that it was fake, whether it was real or a deepfake, and no improvement in telling the two apart was detected. 18 entries are classified as errors (18 Minor), and 1 is a clarity note. Five possible errors were not confirmed, and three remain undetermined.

What was and was not checked

I checked the following results, which held up with the qualifications noted below.

Table 1 and attrition in Study 1 (p. 5). The cell sizes 347, 349, 349 and 351 and the total of 1,396 follow from the Study 1 raw data. The one IP address that was randomized three times is the only duplicate removed, as Section 2 of the Appendix says. For the test of attrition, I find a Pearson statistic of $11.31$ on 3 degrees of freedom, with $p=0.010$ . Attrition is 4.9% in the arm with the real video and no warning, against 1.8% (19 of 1,049) in the other three arms.
The main estimates of Study 1 (pp. 7–8, Figure 2). The means are $-0.408$ and $0.102$ for the deepfake video and $-0.485$ and $0.235$ for the real video. The adjusted effects are $0.499$ and $0.726$ , each with robust standard error $0.116$ . The intervals with Manski bounds, $[0.121,0.855]$ and $[0.288,1.100]$ , agree with Figure 2. Both effects stay significant at the 5% level under the adverse imputation. The estimates change little with the covariates: $0.497$ and $0.719$ with the planned covariates, and $0.511$ and $0.720$ with none. The interaction of the two treatments is $-0.214$ with standard error $0.163$ and $p=0.191$ , as in footnote 17. The printed $p<0.001$ for the deepfake video is the subject of P034-E003.
The response shares on pp. 7–8 agree with Tables OA4a and OA4c and, for Study 1, with the raw data. In Tables OA4a–OA4c all counts, sums, percentages and totals recompute. Cohen’s $d$ in footnotes 14, 16 and 18 recomputes as $0.337$ , $0.480$ and $0.197$ .
Footnote 8 and Table OA2: every count, row percentage and total follows from the raw data, as do the shares 29.9%, 27.1%, 15.8% and 27.2% and Cramer’s $V=0.082$ . The $p$ -value printed in the note to the table is corrected in P034-E015.
Section 3 of the Appendix and Figure OA1: the mean of $-0.48$ for the real video, the adjusted difference of $0.062$ , and the intervals $[-0.384,0.469]$ , $[-0.169,0.187]$ and $[-0.123,0.085]$ for the three outcomes. The wording drawn from these numbers is the subject of P034-E002. The Study 1 panels of Figures OA2a and OA2b and the Study 1 points of Figure OA3 also recompute. Which observations the bounding program replaces is the subject of P034-E005.
Table OA1: the Study 1 column follows from the raw data, and the printed median income band of Study 1 is correct for the data before imputation. The Study 2 and pooled columns agree with the stored workbook. The postgraduate row is the subject of P034-E009.
Table OA3: the column totals are $2{,}898=3\times966$ , $2{,}877=3\times959$ and $5{,}775=3\times1{,}925$ , and the percentages recompute except in two cells (P034-N010). The drop of 3.1 points follows from 49 of 966 against 19 of 959, and 58 responses with pro-choice or pro-life codes in each arm agree with $p=0.95$ ; the outcome description and some hand codes are corrected in P034-E007, and the loss of free-response text before hand-coding in P034-E010.
Tables OA5 and OA6: the printed coefficients and bounds follow from the replication programs. Their labels are the subject of P034-A001. In Tables OA7–OA12 each printed interval is symmetric about its estimate up to rounding, and Tables OA8–OA10 agree with the stored workbook. Tables OA7–OA9, OA11 and OA12 also recompute from the Study 2 data for the samples actually used; the subgroup interpretation is corrected in P034-E001 and departures from the plan in P034-N011. The stored values for Tables OA7, OA11 and OA12 are the subject of P034-E019. In Section 5 of the Appendix, the printed $F(34,1795)=1.75$ corresponds to $p=0.005$ , which agrees with the printed $p<.01$ .
I computed seven headline analyses again from the public data of both studies, with the Stata programs translated into R. The three effects of Figure 2 are $0.499$ , $0.726$ and $0.264$ . The pooled effects are $0.398$ , $0.125$ and $6.0$ percentage points. The effect on naming a pro-choice position is $-3.1$ points, the test for giving a pro-choice or pro-life code has $p=0.95$ , and the effect on the closed question is $-14.1$ points. These agree with the printed values up to rounding; the one difference is the $p$ -value treated in P034-E003.
Design and coding: the 65,000 training iterations are the sum of 50,000 and 15,000 (Section 1 of the Appendix). The warning text in Tables 1 and 2 agrees with the script of Study 2. The restrictions of Study 2 use questions asked before treatment. The final party variable corrects the coding defect that the Appendix acknowledges. The code printed in Section 8 of the Appendix agrees with the replication file, and the closed question is coded as the plan specifies. For the main outcome, the bounding program does what Section 2 of the Appendix describes.
The pre-analysis plans: the target of 362 participants per arm in Study 1 follows from the stated standard deviation and effect. The amendment to the Study 2 plan adds only the last exclusion by date. The rule of that plan for missing data is the subject of P034-E017.

The paper contains no theorem, proof or derivation. The checks had the following limits. Study 1 was rebuilt from its raw file. Beyond the seven headline analyses, I used the public Study 2 data to check free-response text and coding, subgroup and alternative trust effects, and the duplicate-IP and election exclusions. Other supplementary Study 2 results were compared only with the printed tables, the figures and the stored workbooks. The seven headline analyses use an R translation of the authors' programs, and the later Study 2 checks use Python translations rather than native Stata execution; not all supplementary analyses were recalculated. The intended population for counting duplicate IP addresses remains unclear. The two videos of Study 1 were not viewed. For the video of Study 2, I inspected the file information and two frames and read the transcript. Of the works cited, I read only those needed for particular questions: Wittenberg et al. (2021), Vaccari and Chadwick (2020), Pennycook et al. (2020) and the abstract of Clayton et al. (2020) for P034-N001 and P034-N002, and the sources of two reference entries for P034-E018. I could not obtain the 2021 preprint of Barari, Lucas and Munger or the chapter on attrition of Gerber and Green (2012). The other cited works were not consulted, and the returns of the news searches described on p. 2 were not available. The undetermined questions require the approved Study 1 protocol and fielded survey, authenticated question wording for both studies, or a source that I could not obtain. That I list no further corrections is not a finding that the rest of the paper is correct.

Headline claims and related corrections

Adding the deepfake warning above a deepfake video raised the belief that the video was fake by 0.5 points on the 7-point scale ( $p<0.001$ ), from $-0.4$ to $0.1$ , shifting responses toward both uncertainty and agreement (Study 1).
Figure 2, left panel; Table OA4a; article p. 7 (PDF p. 7)
See P034-E003.
The warning increased disbelief in accompanying real political videos in both experiments: by 0.73 points with the real version of the Study 1 video ( $p<0.001$ ) and by 0.26 points with the real video of Romney in Study 2 ( $p<0.001$ ).
Figure 2, middle and right panels; Tables OA4b and OA4c; article p. 8 (PDF p. 8)
Participants could not tell the real video from its deepfake version, with or without the warning, and telling them about deepfakes did not improve their ability to do so (in Study 1 the interaction of the deepfake and information treatments is not significant and has the opposite sign from the ideal).
Figure OA1 and Appendix Section 3; Figure 2, left and middle panels; interaction test on article p. 8 (PDF p. 8) and footnote 17
See P034-E002.
Pooling both studies (real-video conditions, study fixed effect), the warning raised belief that the video was fabricated by 0.40 points ( $p<0.001$ ), raised 'unconvinced' by 0.13 points on a 3-point scale ( $p<0.001$ ) and raised 'don't know' by 6.0 percentage points ( $p=0.001$ ).
article pp. 8–9 (PDF pp. 8–9) and footnote 19; Figures OA2a and OA2b
See P034-E003.
The warning changed what participants associated with Romney: 3.1 percentage points fewer named a pro-choice position in the open-ended question ( $p<0.001$ ), with a 3.2-point effect under the liberal coding of footnote 20, and 14.1 points fewer marked 'true' that he had ever supported women's access to abortion ( $p<0.001$ ).
Table OA3; article p. 9 (PDF p. 9) and footnote 20
See P034-E006, P034-E007, P034-E010, P034-N010.
The effects were large and consistent across both studies.
article p. 3 (PDF p. 3); Figure 2; footnotes 14, 16, 18
Warning the public about deepfakes erodes trust: information about deepfakes can be used to dismiss genuine political footage, and warned voters may come to distrust political video in general, so public warnings warrant care.
Abstract; article pp. 2–3 (PDF pp. 2–3) and the conclusion (pp. 9-11); Figure 2; Table OA3; Figure OA3 and Table OA7 (neither cited in the main text)

Corrections

Minor (18)

Real but inconsequential, for example a misprint that does not change a result.

P034-E001 Minor

The titles of Tables OA8 and OA9 report “increases” among Republicans and among pro-life participants, but the estimated effects of the information within these two groups are slightly negative and not distinguishable from zero; the tables show differences between groups that are positive and not significant.

Appendix, titles of Tables OA8 and OA9, p. 15 (PDF p. 15 of the Appendix), repeated in the contents, p. 1; Section 5, “Pooled”, p. 19 (PDF p. 19 of the Appendix)

As printed

Appendix p. 15: “Table OA8: Study 2 – Republicans appear to have directional (non-significant) increases in favorability when information about deepfakes is added to Romney video”. The table has the columns Vote and Favor. Information*Republican: 0.06, 0.24. 95% Upper Bound: 0.20, 0.53. 95% Lower Bound: $-0.09$ , $-0.06$ .

Appendix p. 15: “Table OA9: Study 2 – Participants who are pro-life appear to have directional (non-significant) increases in favorability when information about deepfakes is added to Romney video”. Information*Pro-Life: 0.02, 0.13. 95% Upper Bound: 0.17, 0.43. 95% Lower Bound: $-0.13$ , $-0.17$ .

Appendix p. 19: “For instance, in Study 2, pro-life Republicans should be less likely to favor Romney if they believe that he was or is pro-choice. And since information about deepfakes increases disbelief that Romney was ever pro-choice, favorability and vote intention levels in the Information condition should be higher than in the No Information condition. This pattern should be reversed for pro-choice Democrats. Our results (see Section 3, Tables 6-7) are consistent with this theory in terms of estimate directions but are not statistically significant.”

The problem

Each table reports, for each outcome, one coefficient with its bounds: the interaction of the information about deepfakes with the group. In each regression, let $b$ be the coefficient on the information and let $d$ be the coefficient on the interaction. The fitted outcome is $a+bT+cG+dTG+x'\theta$ , where $T$ indicates the information, $G$ the group and $x$ the covariates. The estimated effect of the information is then $b$ in the comparison group, which is Democrats in Table OA8 and pro-choice participants in Table OA9, and $b+d$ in the group that the title names. The tables print $d$ and not $b$ . A positive $d$ says that the estimated effect is larger in the named group than in the comparison group. It does not say that the effect in the named group is positive.

In the Study 2 data all four sums $b+d$ are negative. They are $-0.043$ for the vote and $-0.005$ for favorability among Republicans, and $-0.055$ and $-0.049$ among pro-life participants. Each 95% interval for a sum contains zero. The titles describe as increases the two favorability estimates that are slightly negative, and the data establish neither an increase nor a decrease within either group.

Read as a statement about the levels that the theory predicts, the sentence of Section 5 is not supported either: the theory predicts higher favorability and vote intention with the information among pro-life Republicans. The estimates do agree with the prediction of both pre-analysis plans of Study 2, which is comparative. The plans expect the interaction of the treatment with a Republican indicator, and with a pro-life indicator, to be positive. All four interactions are positive, and none is statistically significant. Tables OA8 and OA9 come from separate regressions, and neither estimates a separate effect for respondents who are both pro-life and Republican.

Evidence a recalculation from the source

The evidence is a recalculation from the public Study 2 data. The final cleaning code and the four regressions of the replication files were translated from Stata into Python; Stata was not run. The recalculation gives the analysis sample of 1,925 respondents, 966 without and 959 with information, and it reproduces the four printed interactions with their printed bounds. The regressions of Table OA8 exclude Independents and use 1,750 respondents. Each interval below is the estimate $\pm 1.96$ robust standard errors, as in the Appendix. The standard error of $b+d$ uses the covariance of the two coefficients, that is, $\operatorname{Var}(\hat b+\hat d)=\operatorname{Var}(\hat b)+2\operatorname{Cov}(\hat b,\hat d)+\operatorname{Var}(\hat d)$ .

The coefficient $b$ on the information, the interaction $d$ and the sum $b+d$ in the four regressions of Tables OA8 and OA9. The entries are rounded from the unrounded estimates, so a sum can differ in the last digit from the sum of the rounded coefficients.
Table, outcome	$N$	$b$	$d$	$b+d$	SE of $b+d$	95% interval for $b+d$
OA8, Vote	1,750	$-0.099$	$0.055$	$-0.043$	$0.052$	$[-0.146,\,0.059]$
OA8, Favor	1,750	$-0.243$	$0.238$	$-0.005$	$0.110$	$[-0.221,\,0.211]$
OA9, Vote	1,925	$-0.076$	$0.021$	$-0.055$	$0.061$	$[-0.175,\,0.065]$
OA9, Favor	1,925	$-0.179$	$0.130$	$-0.049$	$0.127$	$[-0.298,\,0.200]$

The vote is coded $-1$ , $0$ , $1$ and favorability from $-3$ to $3$ , and the entries are points on these scales. The signs do not depend on rounding: solving the normal equations of the four regressions in exact rational arithmetic, I find that $b<b+d<0$ and $d>0$ in each.

What survives

The four interactions, their printed bounds and their lack of statistical significance stand, and their signs agree with the predictions of both pre-analysis plans. The regressions are unchanged; only their description changes. The pooled estimates of Section 5 and the results of the main text come from other regressions and do not depend on these titles. What changes is the reading of the tables: the estimates do not show increases within either group, and the tables do not support the levels that the theory of Section 5 predicts. The check does not establish a decrease within either group.

Corrected

Appendix pp. 1 and 15: “Table OA8: Study 2 – Information effects on vote intention and favorability are estimated to be less negative among Republicans than Democrats; the differences are not statistically significant”.

Appendix pp. 1 and 15: “Table OA9: Study 2 – Information effects on vote intention and favorability are estimated to be less negative among pro-life than pro-choice participants; the differences are not statistically significant”.

Appendix p. 19, in place of the last sentence quoted above: “The interaction point estimates of Tables OA8–OA9 are positive, consistent with the pre-specified comparative predictions, but the differences are not statistically significant. The effects within Republicans and within pro-life participants are estimated to be negative on both outcomes, with 95% intervals that include zero; these tables do not estimate a separate effect for participants who are both pro-life and Republican.”

The sentences before it can stay as a statement of the theory. An effect within a group is the sum $b+d$ , and it is to be reported with an interval that uses the covariance of the two coefficients, as in the table above. The cross-reference “Section 3, Tables 6-7” of the printed sentence is corrected in P034-E015.

Crop of Table OA8 with its title: Table OA8: Study 2 - Republicans appear to have directional (non-significant) increases in favorability when information about deepfakes is added to Romney video. Two red boxes mark the words 'increases in favorability', split over two lines. The table has the columns Vote and Favor and three rows: Information*Republican, 0.06 and 0.24; 95 percent Upper Bound, 0.20 and 0.53; 95 percent Lower Bound, minus 0.09 and minus 0.06. — Appendix p. 15 (PDF p. 15 of the Appendix), Table OA8. The marked words of the title report increases in favorability; the table prints only the interaction of the information with the Republican indicator.

Crop of Table OA9 with its title: Table OA9: Study 2 - Participants who are pro-life appear to have directional (non-significant) increases in favorability when information about deepfakes is added to Romney video. A red box marks the words 'increases in favorability'. The table has the columns Vote and Favor and three rows: Information*Pro-Life, 0.02 and 0.13; 95 percent Upper Bound, 0.17 and 0.43; 95 percent Lower Bound, minus 0.13 and minus 0.17. — Appendix p. 15 (PDF p. 15 of the Appendix), Table OA9. The marked words of the title report increases in favorability; the table prints only the interaction of the information with the pro-life indicator.

Crop of seven lines of Section 5 of the Appendix. The text says that pro-life Republicans should be less likely to favor Romney if they believe that he was or is pro-choice, that favorability and vote intention levels in the Information condition should be higher than in the No Information condition, and that this pattern should be reversed for pro-choice Democrats. Two red boxes mark the words 'consistent with this theory in terms of estimate directions', split over two lines, in the sentence 'Our results (see Section 3, Tables 6-7) are consistent with this theory in terms of estimate directions but are not statistically significant.' — Appendix p. 19 (PDF p. 19 of the Appendix), Section 5, “Pooled”. The marked words say that the results agree with the theory in the direction of the estimates; the theory stated before them predicts higher favorability and vote intention with the information.

P034-E017 Minor

The absence of differential attrition does not show that outcomes are missing at random; the complete-case analysis of Study 2 needs a separate assumption about missingness, which the chi-squared test cannot supply.

Amended pre-analysis plan for Study 2, Section 8 (Missing Data), second paragraph (PDF p. 5 of the plan, whose pages are not numbered); notes to Figure OA2a, Appendix p. 9, and to Figure OA3, Appendix p. 18; related sentence in Section 2.2 of the article, p. 6 (PDF p. 6)

See also under Formal statements (Lean): P034-T1, P034-T2

As printed

Plan, Section 8: “Since there may be individuals who are randomized but do not finish the survey (and thus do not provide outcome responses), we will check for differential attrition using a chi-squared test (see line 13 in the Stata code). If there is no differential attrition, we conclude that missingness occurs at random and exclude all affected observations from analysis. If there is differential attrition (i.e., chi-squared p-value $<$ .05), we will use inverse probability weights to weight the population of all subjects who passed the attention checks.”

Appendix p. 9, note to Figure OA2a: “The rightmost treatment effect graph uses covariate-adjusted standard errors from an OLS regression, as there was no differential attrition.”

Appendix p. 18, note to Figure OA3: “The confidence intervals in Study 2 are estimated using covariate-adjusted standard errors from an OLS regression, as there was no differential attrition.”

The problem

The plan treats a chi-squared test that does not reject equal completion rates as showing that outcomes are missing at random. On that basis it analyzes only the respondents who completed the survey. The conclusion does not follow. I write $T$ for the randomized treatment, $Y(0)$ and $Y(1)$ for the potential outcomes, $R(0)$ and $R(1)$ for the indicators that the outcome would be observed under each assignment, and $X$ for the pretreatment covariates. The outcome $Y=Y(T)$ is observed when $R=R(T)$ equals $1$ . Under randomization the test addresses the hypothesis

P(R=1\mid T=0)=P(R=1\mid T=1),

that is, $\mathbb{E}[R(0)]=\mathbb{E}[R(1)]$ . This hypothesis concerns how many outcomes are missing in each arm, and not which outcomes are missing. The complete-case difference in means converges to $\tau_{\mathrm{CC}}=\mathbb{E}[Y(1)\mid R(1)=1]-\mathbb{E}[Y(0)\mid R(0)=1]$ . For the average effect $\tau=\mathbb{E}[Y(1)-Y(0)]$ , I have

\tau_{\mathrm{CC}}-\tau=\bigl\{\mathbb{E}[Y(1)\mid R(1)=1]-\mathbb{E}[Y(1)]\bigr\}-\bigl\{\mathbb{E}[Y(0)\mid R(0)=1]-\mathbb{E}[Y(0)]\bigr\}.

Equal completion rates do not make either bracket vanish. The implication fails even when the two rates are known to be equal, and the gap is therefore not one of statistical power.

The rule of the plan was applied. The analysis program of the replication files tests completion by arm and carries the comment “attrition not differential, so as per PAP we drop missings”. It then drops the respondents with a missing outcome before the Study 2 regressions and the pooled regressions. The notes to Figures OA2a and OA3 say that the intervals of Study 2 are complete-case intervals because there was no differential attrition.

Evidence an exact counterexample

The evidence is an exact counterexample, computed in rational arithmetic. Let $0<b<a<1$ . Let $U$ and $X$ be independent Bernoulli variables with parameter $1/2$ , and set $Y(0)=Y(1)=U$ , so that the effect is zero for every unit. Given $U$ , let $R(0)$ and $R(1)$ be independent with

\begin{aligned} P(R(0)=1\mid U=0)&=a, & P(R(0)=1\mid U=1)&=b,\\ P(R(1)=1\mid U=0)&=b, & P(R(1)=1\mid U=1)&=a, \end{aligned}

and let $T$ be a Bernoulli variable with parameter $1/2$ , independent of all the others. Then the completion rate is $(a+b)/2$ in each arm. The complete-case mean of $Y$ is $b/(a+b)$ in the control arm and $a/(a+b)$ in the treated arm, and the difference is $\delta=(a-b)/(a+b)>0$ . Among the complete cases $X$ is independent of $(T,Y)$ and the two arms are equally likely. Solving the normal equations of the complete-case regression of $Y$ on $(1,T,X)$ , I find that the coefficient on $T$ is also $\delta$ . Two choices of $a$ and $b$ are listed in the table. For each choice the table shows a sample that the process produces with positive probability, and in this sample the chi-squared statistic for completion by arm is exactly zero.

Two cases of the counterexample. The true effect is zero in both. The sample of each column is divided evenly between $X=0$ and $X=1$ in every cell.
	First case	Second case
$a$ , $b$	$37/50$ , $13/50$	$499/500$ , $489/500$
Attrition in each arm	$1/2$	$3/250$ ( $1.2\%$ )
Complete-case mean of $Y$ , control and treated	$13/50$ , $37/50$	$489/988$ , $499/988$
Complete-case coefficient on $T$	$12/25=0.48$	$5/494\approx0.0101$
Sample size	$400$	$4000$
Missing and observed in each arm	$100$ , $100$	$24$ , $1976$
Observed $Y=0$ and $Y=1$ , control	$74$ , $26$	$998$ , $978$
Observed $Y=0$ and $Y=1$ , treated	$26$ , $74$	$978$ , $998$
Pearson chi-squared statistic, $p$ -value	$0$ , $1$	$0$ , $1$

In both cases the distribution of the observed data, that is, of $(T,X,R)$ and of $Y$ when $R=1$ , is the same as under a second process in which outcomes are missing completely at random and the true effect is $\delta$ . This implies that no test computed from the observed data can tell the two processes apart.

The corrected statement below has two hypotheses: (i) the treatment is randomized, and (ii) completion under each treatment is independent of the potential outcomes and the covariates and has positive probability. It holds for the following reason. By (i), the event $\{T=t,R=1\}$ equals $\{T=t,R(t)=1\}$ and has probability $P(T=t)P(R(t)=1)>0$ . By (i) and (ii), the law of $(Y(t),X)$ given this event is its unconditional law. This implies the first two conclusions. Let $p=P(T=1\mid R=1)$ , and let $(\beta_0,\beta_T,\beta_X)$ solve the normal equations. Subtracting $p$ times the equation for the intercept from the equation for $T$ , I obtain

\mathbb{E}\bigl[(T-p)(Y-\beta_0-\beta_TT-\beta_X'X)\mid R=1\bigr]=0.

Note that $\mathbb{E}[(T-p)X\mid R=1]=0$ , since $X$ has the same law in both arms of the complete cases, and that $\mathbb{E}[(T-p)T\mid R=1]=p(1-p)$ . The display then reads $p(1-p)\{\mathbb{E}[Y\mid T=1,R=1]-\mathbb{E}[Y\mid T=0,R=1]\}=\beta_T\,p(1-p)$ , and I conclude that $\beta_T=\tau$ .

What survives

The sentence of the article on p. 6 reports the result of a test, and it stands; I did not repeat the test for this finding. The randomized design, the arithmetic of the regressions and the comparisons among respondents who completed the survey are unaffected. The bounds used for Study 1 do not rely on this inference. No bias, wrong estimate, wrong $p$ -value or failed interval in Study 2 is established. What changes is the justification. The Study 2 and pooled estimates describe the randomized sample only under an additional assumption about missingness, and the test cannot supply it. If the assumption is stated, it is a new hypothesis that the data may not satisfy. If bounds replace it, the conclusions may become weaker.

Corrected

Plan, Section 8: “If the chi-squared test does not reject, we will report no detected difference in completion rates across treatment arms. This result does not establish random or ignorable outcome missingness. Complete-case covariate-adjusted OLS requires a separate identifying condition; for example, within the target population, completion under each treatment is independent of potential outcomes and pretreatment covariates, with positive completion probability. The test does not verify that condition. If a suitable identifying condition cannot be justified, we will report attrition bounds or sensitivity analyses.”

In the notes to Figures OA2a and OA3, the clause “as there was no differential attrition” should be replaced in the same way. The intervals of Study 2 are complete-case intervals, no difference in completion rates was detected, and reading them as intervals for the randomized sample rests on the separate condition.

In full, one sufficient condition for the complete-case analysis is the following. Let $T\in\{0,1\}$ be the treatment, let $Y(0)$ and $Y(1)$ be real potential outcomes with finite second moments, let $R(0),R(1)\in\{0,1\}$ be the potential completion indicators, and let $X$ be a vector of pretreatment covariates with finite second moments. Set $Y=Y(T)$ , $R=R(T)$ and $\tau=\mathbb{E}[Y(1)-Y(0)]$ . Suppose that (i) $T$ is independent of $(Y(0),Y(1),R(0),R(1),X)$ and $0<P(T=1)<1$ , and that (ii) for each $t\in\{0,1\}$ , $R(t)$ is independent of $(Y(0),Y(1),X)$ and $P(R(t)=1)>0$ . Then $\mathbb{E}[Y\mid T=t,R=1]=\mathbb{E}[Y(t)]$ for $t=0,1$ , the law of $X$ given $\{T=t,R=1\}$ is the law of $X$ , and every solution of the normal equations of the complete-case regression of $Y$ on $(1,T,X)$ has coefficient $\tau$ on $T$ . Condition (ii) does not follow from (i) and equal completion rates. In the first case of the table, (i) holds and both completion rates are $1/2$ , while $\tau=0$ and the coefficient on $T$ is $12/25$ .

Crop of six lines of the plan. Two red boxes mark the words 'we conclude that missingness occurs at random', split over two lines, in the sentence 'If there is no differential attrition, we conclude that missingness occurs at random and exclude all affected observations from analysis.' — Amended pre-analysis plan for Study 2, Section 8, second paragraph (PDF p. 5 of the plan). The marked words draw the conclusion from the absence of differential attrition.

Crop of the last four lines of the note to Figure OA2a. Two red boxes mark the words 'as there was no differential attrition.', split over two lines, in the sentence 'The rightmost treatment effect graph uses covariate-adjusted standard errors from an OLS regression, as there was no differential attrition.' — Appendix p. 9 (PDF p. 9 of the Appendix), note to Figure OA2a. The marked clause gives the absence of differential attrition as the reason for the complete-case interval of Study 2.

Crop of four lines of the note to Figure OA3. A red box marks the words 'as there was no differential attrition.' in the sentence 'The confidence intervals in Study 2 are estimated using covariate-adjusted standard errors from an OLS regression, as there was no differential attrition.' — Appendix p. 18 (PDF p. 18 of the Appendix), note to Figure OA3. The marked clause gives the same reason.

Crop of the last four lines of Section 2.2. A red box marks the words 'we found no evidence of differential attrition:' in the sentence 'In Study 2, we found no evidence of differential attrition: survey completion rates did not differ significantly across treatment arms.' — Section 2.2 of the article, p. 6 (PDF p. 6). The marked sentence reports the result of the test. It is not in error; the inference from it is made in the plan and in the two notes of the Appendix.

P034-E006 Minor

The replication code for the robustness check of footnote 20 keeps the conservative codes and changes the covariate list, so it does not carry out the check that the footnote describes; a recalculation with the liberal codes and Study 2 covariates reproduces the printed 3.2 percentage points (see `P034-E010`).

Footnote 20, p. 9 (PDF p. 9); replication code, 3-analysis_FINAL.do, lines 393–413, with lines 227–228, 296, 317 and 366

As printed

Footnote 20, p. 9. We report the most conservative coding so that ambiguous responses like “support women” and “fight for women’s causes” are coded as “other.” As a robustness check, we conduct the same analysis but with a liberal interpretation of the same responses (i.e., we code these types of ambiguous responses as “pro-choice”). Using this liberal interpretation, we recover a slightly larger treatment effect of 3.2 percentage points ( $p<0.001$ ).

3-analysis_FINAL.do, lines 399 and 409.

merge 1:1 responseid using […], gen(_merge_handcode_lib)

reg FR_abortion_flag tr_warn $covars if study==2 & romney_party==1, robust

The problem

Footnote 20 describes the analysis behind the estimate of 3.1 percentage points, repeated with one change: ambiguous responses are coded as pro-choice. The block of the replication code that produces the regression of the footnote does not do this, for two reasons. First, the conservative code variables final_code1, final_code2 and final_code3 are merged into the analysis data at line 296 and are never dropped. The block writes the liberal codes to a file and merges that file at line 399 with a plain one-to-one merge. Under Stata’s default rule for a merge, a matched observation keeps the values already in memory for every variable that both files contain. The liberal codes therefore never enter the data, and the new outcome equals the conservative outcome on every row. Second, the regression at line 409 takes its covariates from the global macro covars. Line 366 has reset this macro to the six covariates of Study 1. The conservative regression at line 317 used the expanded list of Study 2 that lines 227–228 define. The block thus fits the conservative outcome with a different covariate list, and its output cannot show what the liberal coding does to the estimate.

Evidence a recalculation from the source

The evidence is a trace of the code and an exact check on the file of hand-coded responses in the replication files, which has 7,671 responses from 2,557 participants. Applying the conservative rules to this file, I find 103 pro-choice responses from 101 participants; the liberal rules give 138 from 134. The two sets of codes differ for 36 responses from 36 participants, and the outcome would change for 33 participants; with the merge at line 399 it changes for none. For example, one participant names the facts “MAN”, “WHITE” and “FOR WOMEN’S RIGHTS”. The outcome variable FR_abortion_flag equals $1$ when no response has a pro-choice code. With the conservative codes demog, demog and other of this participant, it equals $1$ . With the liberal codes demog, demog and curr_prochoice, it equals $0$ , and the block returns $1$ . The counts above refer to the whole file and not to the regression sample. I checked the documented default merge rule without running Stata. In the recalculation reported in P034-E010, I generated the conservative and liberal outcomes separately and matched them to the Study 2 sample, avoiding the merge that retains the conservative codes. With the liberal outcome, the expanded Study 2 covariates and HC1 standard errors, I obtain a drop of 3.20 percentage points ( $p=0.0006$ ) on $N=1{,}925$ . The calculation translates the relevant Stata regression into Python and uses the supplied cleaned covariates; Stata was not run.

What survives

The conservative estimate of 3.1 percentage points at line 317, the closed-ended result of 14.1 percentage points and the main analyses of both studies are computed before this block and do not depend on it. I reproduce the 3.2 percentage points and $p<0.001$ of footnote 20 in the translated calculation of P034-E010. The deposited replication code still needs both repairs to carry out the stated robustness check.

Corrected

3-analysis_FINAL.do, lines 398–409. The conservative code variables are dropped before the second merge, and the regression names the covariates of Study 2:

restore

drop final_code1 final_code2 final_code3

merge 1:1 responseid using […], gen(_merge_handcode_lib)

Lines 401–407, which construct FR_abortion_flag from the merged codes, remain between the merge and the regression:

reg FR_abortion_flag tr_warn c.age i.gender i.party i.region c.hhi ///

elder-pro_vax conlib imputed_conlib bigfive* pol_know_total polinterest ///

if study==2 & romney_party==1, robust

Footnote 20. I reproduce the printed estimate and $p$ -value in the translated calculation of P034-E010. The repaired Stata code should be run on the Study 2 analysis data to verify its output against that calculation.

Crop of the six lines of footnote 20. The footnote says that the most conservative coding is reported, that as a robustness check the same analysis is conducted with a liberal interpretation of the same responses, and continues: Using this liberal interpretation, we recover a slightly larger treatment effect of 3.2 percentage points (p less than 0.001). A red outline surrounds the words 3.2 percentage points (p less than 0.001). The last sentence points to Section 8 of the Appendix for the full list of ambiguous responses. — Footnote 20, p. 9 (PDF p. 9). The outline marks the estimate and $p$ -value reported for the liberal coding.

Listing of sixteen numbered lines of Stata code under the heading ORIGINAL CODE: 3-analysis_FINAL.do, SOURCE LINES 394-409. Line 394 renames FR_abortion_flag to FR_abortion_flag_conservative. Lines 395 to 398 preserve the data, set the global frmethod to liberal, run free_response_cleanup.do and restore. Line 399 reads merge 1:1 responseid using data/study2/study2_coded_free_responses_to_merge, gen(_merge_handcode_lib). Lines 401 to 407 generate FR_abortion_flag equal to 1 and replace it by 0 if any of the three final codes is curr_prochoice or past_prochoice. Line 409 regresses FR_abortion_flag on tr_warn and the macro covars for Study 2 participants with romney_party equal to 1. Nothing is outlined. — Replication code, `3-analysis_FINAL.do`, lines 394–409. Line 399 merges the liberal codes without replacing the conservative ones already in memory, and line 409 runs the regression with the macro `covars`.

P034-E007 Minor

Two hand-coded responses that support abortion choice are coded as pro-life, and the outcome described as mentioning abortion is an indicator of a pro-choice or pro-life code, which misses six participants who name abortion.

Section 3, second paragraph of p. 9 (PDF p. 9); Appendix, Table OA3, p. 11, and Section 8, p. 23; file of hand-coded responses in the replication files; 3-analysis_FINAL.do, lines 302–313

As printed

Section 3, p. 9. We found that information about deepfakes did not significantly change how likely participants were to mention abortion ( $p=0.95$ ), but there was a major shift in what participants perceived Romney’s position on abortion to be. The information about deepfakes caused a 3.1-percentage-point drop in the percentage of participants who associated Romney with a pro-choice position ( $p<0.001$ ).

Hand-coded responses in the replication files. “HE WILL FIGHT FOR WOMEN’S RIGHTS TO CHOOSE”: curr_prolife. “HE’S PRO ABORTION”: curr_prolife.

The problem

The two analyses in this passage use hand codes for the three facts that each participant named. First, two responses that state support for abortion choice carry the code for a current pro-life position. With the correct code, each of the two participants counts as associating Romney with a pro-choice position, which bears on the estimate of 3.1 percentage points and on the pro-choice and pro-life rows of Table OA3. These two responses are not the only ones: the response “HE WILL LET WOMEN HAVE THE RIGHT TO CHOOSE FOR ABORTION” has the same reversed code. Second, the outcome behind $p = 0.95$ does not measure whether a participant mentions abortion. The variable mention_abort equals $1$ exactly when at least one of the three responses has a current or past pro-choice code or a current or past pro-life code. A response that names abortion and receives another code does not count. The conservative cleanup printed on p. 23 of the Appendix assigns the code “other” to the bare response “ABORTION”. Six participants in the file of hand-coded responses name abortion and have mention_abort equal to $0$ .

Evidence a recalculation from the source

The evidence is an exact check on the file of hand-coded responses, which has 7,671 responses from 2,557 participants, and a derivation from the code. For $j = 1, 2, 3$ , I set $P_j = 1$ if response $j$ has a current or past pro-choice code and $P_j = 0$ otherwise, and I define $L_j$ in the same way for the pro-life codes. Lines 302–311 of 3-analysis_FINAL.do start with FR_abortion_flag equal to $1$ and mention_abort equal to $0$ . For each $j$ in turn, they set the first variable to $0$ if $P_j = 1$ , and then set the second to $1$ if $L_j = 1$ or the first variable is $0$ . I write $F$ and $M$ for the final values of the two variables. By induction on $j$ , I obtain

F = \prod_{j=1}^{3} (1 - P_j), \qquad M = 1 - \prod_{j=1}^{3} (1 - P_j - L_j) .

The second equation shows that $M$ indicates a pro-choice or pro-life code, and not a mention of the topic. One participant’s responses have the codes other, dontknow and dontknow, and the first response is the word “ABORTION”. Since $P_j = L_j = 0$ for every $j$ , I have $M = 0$ for this participant. The six participants who name abortion and have $M = 0$ gave the responses “ABORTION” (four participants), “MADE CLAIMS ABOUT ABORTION” and “MORMON CHURCH DOES NOT SUPPORT ABORTION”.

In the whole file, the two recodes raise the number of participants with a pro-choice code from 101 to 103 and leave $M$ unchanged. There are 150 participants with $M = 1$ , and counting the six participants as well gives 156. The 53 rules of the conservative cleanup, applied to the file, reproduce all 7,671 codes saved in the replication files. The file covers 2,557 participants, and the analysis sample has 1,925. I did not establish analysis-sample membership for all of these participants or compute the corrected Table OA3 and regressions.

What survives

The two recodes do not change mention_abort, and describing this variable accurately changes no number. The result $p = 0.95$ stands as a statement about pro-choice and pro-life codes. The outcomes for disbelief in the video and the closed-ended result of 14.1 percentage points use other variables. The estimate of 3.1 percentage points, its $p$ -value and the pro-choice and pro-life rows of Table OA3 are to be recomputed after the recodes. The recodes concern a few responses, and their effect on these numbers was not computed.

Corrected

Section 3, p. 9. We found that information about deepfakes did not significantly change how likely participants were to give a response coded as a current or past pro-choice or pro-life position ( $p=0.95$ ), but there was a major shift in what participants perceived Romney’s position on abortion to be.

Hand-coded responses. “HE WILL FIGHT FOR WOMEN’S RIGHTS TO CHOOSE”: curr_prochoice. “HE’S PRO ABORTION”: curr_prochoice. The response “HE WILL LET WOMEN HAVE THE RIGHT TO CHOOSE FOR ABORTION” carries the same reversed code and is to be recoded with them; the hand codes were not reviewed exhaustively. Table OA3 and the estimate of 3.1 percentage points are then to be recomputed on the analysis sample.

Crop of nine lines of the article, beginning: We also evaluated whether information about deepfakes can affect how participants internalize the content of the video. The marked sentence reads: We found that information about deepfakes did not significantly change how likely participants were to mention abortion (p equals 0.95), but there was a major shift in what participants perceived Romney's position on abortion to be. A red outline surrounds the words mention abortion (p equals 0.95). The following sentence says that the information about deepfakes caused a 3.1-percentage-point drop in the percentage of participants who associated Romney with a pro-choice position (p less than 0.001). — Section 3, second paragraph of p. 9 (PDF p. 9). The outline marks the outcome described as mentioning abortion, with its $p$ -value of 0.95; the next sentence gives the drop of 3.1 percentage points.

Listing of fourteen numbered lines of Stata code under the heading ORIGINAL CODE: 3-analysis_FINAL.do, SOURCE LINES 302-315. Line 302 generates FR_abortion_flag equal to 1 and line 304 generates mention_abort equal to 0. A loop over i from 1 to 3 replaces FR_abortion_flag by 0 if final_code i equals curr_prochoice or past_prochoice, and replaces mention_abort by 1 if final_code i equals curr_prolife or past_prolife or FR_abortion_flag equals 0. Line 313 regresses mention_abort on tr_warn and the covariates for Study 2 participants with romney_party equal to 1, and the comment on lines 314 and 315 quotes the sentence of page 9 with p equals 0.95. Nothing is outlined. — Replication code, `3-analysis_FINAL.do`, lines 302–315. Lines 304 and 309–310 build `mention_abort` from the pro-choice and pro-life codes, and line 313 is the regression behind $p = 0.95$ .

P034-E010 Minor

Six free-response answers from five respondents of Study 2 were lost or cut short before they were hand-coded; one substantive answer in the analysis sample was coded as “don’t know”, and the count 777 in the first row of Table OA3 should be 776, with total 1,594.

Appendix, Table OA3, first row, p. 11 (PDF p. 11 of the Appendix); the paragraph on the open-ended question and footnote 20, p. 9 (PDF p. 9); replication files: the raw survey file of Study 2, the cleaned data, the file of hand-coded responses, and 2-study2_data_cleanup_FINAL.do, line 2

As printed

Appendix p. 11, “Table OA3: Study 2 - Hand-coded Romney facts across conditions”, first row, under the columns “No Info about Deepfakes”, “Info about Deepfakes” and “Total”.

Don’t Know / No Response Given: 28.2%, 27.0%, 27.6%. $N$ : 818, 777, 1,595.

File of hand-coded responses in the replication files, third answer of one respondent in the arm with information about deepfakes: the text is empty, and the code is dontknow.

The problem

Each participant of Study 2 was asked to name three facts about Romney, and each answer was coded by hand. The codes are the basis of Table OA3 and of the analyses of the open-ended question on p. 9. Some answers did not reach the coders as they were written. Among the answers of the 2,991 respondents in the cleaned data, seven contain a double quotation mark in the raw survey file. In the cleaned data, each of the seven consists only of the text after its last quotation mark. Six of the seven are among the 7,671 answers that were hand-coded, and they come from five respondents. Five of these entries are empty, and the sixth reads “FOR OBAMACARE” in place of a full sentence.

All five empty entries have the code dontknow. For three of them the code is right, since the respondent wrote “don’t know” between quotation marks. The other two respondents gave substantive answers, He asked to be called "Ike" and Canceled "Big Bird". The first of these two respondents did not identify Romney’s party and is outside the analysis sample. The second is in the analysis sample, in the arm with information about deepfakes. The first row of Table OA3 counts this answer as “Don’t Know / No Response Given”, and its entries 777 and 1,595 are too large by one.

Evidence a recalculation from the source

The evidence is a recalculation from the replication files. I compared the raw survey file of Study 2 with the cleaned data and with the file of hand-coded responses, matching the answers by respondent and by their position among the three. The cleaned data hold 8,973 answers from 2,991 respondents, and seven differ from the raw file. The file of hand-coded responses holds 7,671 answers from 2,557 respondents, and six differ. The seventh belongs to a respondent who fails the age check and was not hand-coded. Every other answer agrees with the raw file up to capitalization and spacing. The six answers are the following.

He asked to be called "Ike". The hand-coded entry is empty, with the code dontknow. The respondent is outside the analysis sample.
'don't know". The entry is empty, with the code dontknow. The respondent is in the analysis sample, in the arm with information.
Canceled "Big Bird". The entry is empty, with the code dontknow. The respondent is in the analysis sample, in the arm with information.
" don't know ", written twice by one respondent, as the second and the third answer. Both entries are empty, with the code dontknow. The respondent is in the analysis sample, in the arm without information.
A sentence on the health care plan that Romney supported in Massachusetts, which ends which was said to be the "model" for Obamacare. The entry is “FOR OBAMACARE”, with the code policy. The respondent is in the analysis sample, in the arm with information.

With the codes as they are in the replication files, I reproduce every count of Table OA3. The answer of item 3 is substantive, and the table changes as follows when it leaves the first row. Which substantive row receives it is a coding judgment, and the table gives two choices.

The rows of Table OA3 that change when the answer of item 3 is coded as substantive. Each entry gives the counts for the columns “No Info about Deepfakes”, “Info about Deepfakes” and “Total”.
Row	Printed	After the answer is restored
Don’t Know / No Response Given	818, 777, 1,595	818, 776, 1,594
Other, if the answer is coded `other`	258, 240, 498	258, 241, 499
Stance on Other Policies, if it is coded `policy`	24, 25, 49	24, 26, 50

The denominators 2,898, 2,877 and 5,775 do not change, and the percentages of the first row still round to 28.2%, 27.0% and 27.6%. With the code other, the percentage of the row “Other” in the column with information becomes 8.4% (printed 8.3%).

None of the six answers has a pro-choice or pro-life code, and none receives one when its text is restored. This implies that the two outcomes that the analyses of p. 9 build from the hand codes are the same before and after for each of the 2,557 respondents. This holds under both the conservative coding and the liberal coding of footnote 20. I read the files with a strict parser of comma-separated text in Python and repeated the comparison in R. Stata was not run. The check does not identify what in the import of the raw file (insheet, line 2 of 2-study2_data_cleanup_FINAL.do) causes the loss.

What survives

Restoring these responses changes no reported effect. The result $p=0.95$ and the drop of 3.1 percentage points on p. 9, and the drop of 3.2 percentage points in footnote 20, use outcomes that are identical before and after the six answers are restored. In the recalculation these are $p=0.953$ , 3.09 points ( $p=0.0003$ ) and 3.20 points ( $p=0.0006$ ). The denominators of Table OA3 and its rows for pro-choice and pro-life positions are not affected by this correction. Study 1, the closed-ended outcomes of Study 2 and the pooled analyses do not use the hand codes. What changes is one count and one total in the first row of Table OA3, the row that receives the restored answer, and the text of seven responses in the cleaned data, including six in the file of hand-coded responses. Other corrections to Table OA3 are given in P034-E007 and P034-N010.

Corrected

Appendix p. 11, Table OA3, first row.

Don’t Know / No Response Given: 28.2%, 27.0%, 27.6%. $N$ : 818, 776, 1,594.

The answer Canceled "Big Bird" is counted in a substantive row. With the code other, the row “Other” reads 8.9%, 8.4%, 8.6% and $N$ : 258, 241, 499.

In the replication files, the seven damaged responses in the cleaned data and the six corresponding entries in the file of hand-coded responses are to be restored from the raw survey file, matched by respondent and by the position of the answer. The six hand-coded entries are to be coded again from the full text. Three keep the code dontknow, the sentence on health care keeps the code policy, and the two substantive answers receive substantive non-abortion codes. The raw text should be kept beside any normalized text that is used for coding. The conservative and liberal hand-code files are then to be generated separately, followed by Table OA3 and the analyses of the open-ended question.

Crop of the top of Table OA3, titled Study 2 - Hand-coded Romney facts across conditions, with the columns No Info about Deepfakes, Info about Deepfakes and Total. The first row, Don't Know / No Response Given, has the percentages 28.2, 27.0 and 27.6 percent and, below them, the counts N 818, 777 and 1,595. Red boxes mark the counts 777 and 1,595. The second row, Is Republican, has 16.5 percent in each column and the counts 477, 475 and 952. — Appendix, Table OA3, p. 11 (PDF p. 11 of the Appendix): the head of the table and its first two rows. The marked counts, 777 and 1,595, include one substantive answer whose text was lost before hand-coding.

P034-E009 Minor

The stored count behind the Study 1 postgraduate share in Table OA1 includes five respondents whose education is missing (243 for 238); the printed 17% is unchanged, and the pooled 22% is not shown to be wrong.

Appendix, Table OA1 (Summary statistics), p. 7 (PDF p. 7 of the Appendix), row “Postgrad”; replication files, tables/sum_stats_table.xlsx

As printed

Table OA1, Education block, last row, and the last row of the table.
	Study 1	Study 2	Pooled
Postgrad	17%	25%	22%
Total N	1396	1925	3321

For this row, the workbook of the replication files stores the shares $\colorbox{#ffe58a}{$0.1740687679$}$ (Study 1), $0.2493506494$ (Study 2) and $\colorbox{#ffe58a}{$0.2177055104$}$ (pooled).

The problem

The stored share for Study 1 equals $243/1396$ . In the Study 1 raw data, after the final exclusions, there are 1,396 randomized respondents, of whom 238 have a postgraduate education (codes 7 and 8 of the education variable) and 5 have no recorded education. The stored count is $238+5$ : the five respondents with missing education are counted as postgraduates. The stored pooled share equals $723/3321$ , and the stored share for Study 2 equals $480/1925$ . Since $723=243+480$ , the pooled count carries the same five respondents.

The printed entry for Study 1 does not change: $238/1396=17.05\%$ , and $238/1391=17.11\%$ if the five are also removed from the denominator. Whether the pooled entry changes cannot be decided from the Study 1 data. Removing the five respondents, I obtain $718/3321=21.62\%$ , which still rounds to 22%. The pooled entry would round to 21% only if at least four of the 480 respondents counted for Study 2 also had missing education. I did not check this education count against the Study 2 data. The program for the supplementary materials calls a second program to write the workbook, and that second program is not among the replication files. The step that produced the count is therefore not identified.

Evidence a recalculation from the source

The evidence is a recalculation from the Study 1 raw data and from the stored workbook. With the final rules for eligibility, attention checks, duplicates and randomization, the counts of the education codes 1 to 8 are 41, 217, 57, 300, 155, 383, 218 and 20, and 5 respondents have no code. These sum to 1,396, and $218+20=238$ . Multiplying the stored shares by the totals 1396, 1925 and 3321, I obtain 243, 480 and 723.

What survives

The printed 17% for Study 1 is unchanged, and the printed pooled 22% is not shown to be wrong. No treatment effect or conclusion of the paper is shown to depend on the postgraduate count.

Corrected

The row should be defined and its denominator stated: “Postgrad” is the share of all participants with education code 7 or 8, and a missing response is not counted in the numerator. For Study 1 the count is 238 of 1,396, and not 243, and the entry stays at 17%. The entries for Study 2 and for the pooled sample should be recomputed from the Study 2 data with the same definition before the printed 25% and 22% are kept or changed.

Crop of the Education and Partisanship blocks of Table OA1, with three columns of percentages. A red box marks the row 'Postgrad 17 percent, 25 percent, 22 percent'. The rows above read 'HS or less 18, 16, 17', 'Some College 37, 29, 32' and 'Bachelor's Degree 27, 29, 29'; the rows below give partisanship. — Appendix p. 7 (PDF p. 7 of the Appendix), Table OA1, the rows for education and partisanship; the columns are Study 1, Study 2 and Pooled. The marked row is the postgraduate share, whose stored count for Study 1 includes five respondents with missing education.

P034-E011 Minor

The binary vote outcome behind the pooled estimate of 3.8 percentage points gives the value 0 to participants who did not answer the vote question; it should be defined only for participants who answered, which removes 11 participants from the Study 1 regression and moves its coefficient from about $-0.045$ to about $-0.052$ .

Replication file 4-Supplementary-Materials_FINAL.do, line 854; its printed result is in the online Appendix, Section 5, “Pooled”, second sentence, p. 19 (Appendix PDF p. 19)

As printed

Online Appendix, p. 19. Overall, information about deepfakes decreases intention to vote for the politician by 3.8 percentage points (p $<$ 0.05), with little difference in effect sizes across studies.

Replication file 4-Supplementary-Materials_FINAL.do, lines 853–856 (lines 855 and 856 are each one line of code, broken here before imputed_conlib).

\begin{aligned} &\texttt{keep if (study==1 \& tr\_d==0 ) | (study==2 \& romney\_party==1)}\\ &\texttt{gen vote\_bin=}\colorbox{#ffe58a}{$\displaystyle \texttt{vote==1}$}\\ &\texttt{reg vote\_bin tr\_warn~ pro\_vax acupuncture conlib}\\ &\qquad\texttt{imputed\_conlib i.party hhi i.study, robust}\\ &\texttt{bysort study:\ reg vote\_bin tr\_warn~ pro\_vax acupuncture conlib}\\ &\qquad\texttt{imputed\_conlib i.party hhi, robust} \end{aligned}

The problem

The vote question has three answers, coded $-1$ for “No”, $0$ for “Don’t Know / Not Sure” and $1$ for “Yes” (Appendix p. 15). The cleaning code leaves the variable vote missing when the question was not answered. In Stata a missing value is not equal to $1$ , which implies that the expression vote==1 evaluates to $0$ when vote is missing. Line 854 then gives a participant who did not answer the same value as a participant who answered “No” or “Don’t Know / Not Sure”. A missing outcome would be dropped from a regression, and a value of $0$ is kept. The regressions of lines 855 and 856 estimate the effect of information about deepfakes on an outcome that equals $1$ when a “Yes” was recorded and $0$ otherwise, while the sentence on p. 19 reports a decrease in the intention to vote for the politician.

The case occurs in the data. In the two real-video arms of Study 1, 11 participants did not answer the vote question, 9 of 347 in the No Information arm and 2 of 349 in the Information about Deepfakes arm. All 11 enter the regression with the value $0$ . If the outcome is defined only for participants who answered, the Study 1 regression has 685 observations in place of 696, and its coefficient on the information changes from about $-0.045$ to about $-0.052$ . The corrected pooled estimate is not given here. I did not recalculate the pooled regression of line 855 or its Study 2 component. The estimate of 3.8 percentage points and its significance at the 5% level are neither reproduced nor replaced.

Evidence a recalculation from the source

The evidence is a reading of the code and an exact recalculation from the Study 1 raw data, in which I applied the exclusions and imputations of the cleaning code.

Answers to the vote question in the two real-video arms of Study 1.
Arm	Yes	No	Don’t know	Unanswered	All	Answered
No Information	99	158	81	9	347	338
Information about Deepfakes	81	183	83	2	349	347

Without covariates, the difference between the two arms in the share of “Yes” is

\frac{81}{349}-\frac{99}{347} = -\frac{6444}{121103} \approx -0.0532

with the outcome as coded, and

\frac{81}{347}-\frac{99}{338} = -\frac{6975}{117286} \approx -0.0595

among the participants who answered. For the Study 1 regression of line 856, with the covariates of the code and robust (HC1) standard errors, I obtain the following.

The Study 1 regression of line 856 under the two definitions of the outcome.
Outcome	$N$	Coefficient on the information	Robust s.e.	$p$
As coded (unanswered set to 0)	696	$-0.04486$	$0.03289$	$0.173$
Participants who answered	685	$-0.05245$	$0.03326$	$0.115$

The coefficients are computed in exact rational arithmetic, and the standard errors and $p$ -values in floating point. The calculation does not imitate the single-precision storage of Stata, and the last digits may differ from Stata’s output. The pooled regression of line 855 and its Study 2 component were not recalculated.

What survives

The correction touches the binary vote regressions of lines 855 and 856 and the sentence on p. 19 that reports them. The favorability regressions that follow, the analyses of the three-level vote outcome with bounds in Tables OA5 and OA6, and the results of the main text on disbelief, trust and policy knowledge do not use this outcome. Whether the pooled estimate stays near 3.8 percentage points and significant at the 5% level is not established; in Study 1 the coefficient moves away from zero.

Corrected

Line 854 should read

gen vote_bin=(vote==1) if vote<.

The condition if !missing(vote) is equivalent. The outcome vote_bin then equals $1$ for “Yes” and $0$ for “No” and “Don’t Know / Not Sure”, and it is missing when the question was not answered. The regressions of lines 855 and 856 then use the participants who answered. For Study 1, the regression of line 856 has $N = 685$ and a coefficient on the information of about $-0.052$ (robust standard error $0.033$ ), in place of $N = 696$ and about $-0.045$ .

The sentence on p. 19 should report the coefficient of the pooled regression of line 855, computed with this outcome on the data of both studies, and should describe it as an effect among the participants who answered the vote question. The corrected line keeps unanswered questions missing but does not address attrition, and a statement about all randomized participants needs a separate argument.

Crop of the first lines under the heading Pooled. The text reads: Information about deepfakes appears to have some impact on favorability and intention to vote for the politician featured in the video clip. Overall, information about deepfakes decreases intention to vote for the politician by 3.8 percentage points (p<0.05), with little difference in effect sizes across studies. Similarly, information about deepfakes decreases the politician's favorability by 0.17 points on a 7-point scale (from extremely unfavorable to extremely favorable) (p<0.05). The crop ends in the middle of the next sentence, which begins There are, however, strong theoretical reasons. A red outline surrounds the words 3.8 percentage points (p<0.05). — Online Appendix, Section 5, “Pooled”, p. 19 (Appendix PDF p. 19). The outline marks the estimate that the binary vote regression produces.

Listing of six lines of Stata code with line numbers 851 to 856. Line 851 is preserve, line 852 is blank, and line 853 is a keep if command with the condition study==1 and tr_d==0, or study==2 and romney_party==1. A red outline surrounds line 854, gen vote_bin=vote==1. Lines 855 and 856 regress vote_bin on tr_warn and the covariates pro_vax, acupuncture, conlib, imputed_conlib, i.party and hhi with robust standard errors, first pooled with i.study and then by study. — Source code of the replication files, not a page of the article or of the Appendix: `4-Supplementary-Materials_FINAL.do`, lines 851–856. The outline marks line 854, which sets the outcome to 0 when the vote question was not answered.

P034-E012 Minor

Figures OA2a and OA2b show Studies 1 and 2, the note to Figure OA2b should define its binary outcome, the trust item ranges from 0 to 3, and the pooled estimate in Figure OA3 excludes the deepfake arm of Study 1.

Appendix: title of Figure OA2a, p. 9; title and note of Figure OA2b, p. 10; note to Figure OA3, p. 18; the two titles as repeated in the contents, p. 1; Study 1 pre-analysis plan, p. 4

As printed

Figure OA2a, title, p. 9. Figure OA2a: Study 2 - Impacts of information about deepfakes on disbelief (3-point scale): unconvinced that politician believes what is being said

Figure OA2b, title and note, p. 10. Figure OA2b: Study 2 - Impacts of information about deepfakes on disbelief (binary): don’t know if politician truly believes what is being said

[…] All three columns measure impact in terms of the same outcome: how convinced are you that the politician believes what is being said? (3-point scale from 1=”not at all convinced” to -1=”very convinced”).

Figure OA3, note, p. 18. Responses range from 0 = “not at all confident” to 4 = “completely confident.” […] The pooled estimate is estimated using a regression with shared covariates; it excludes all participants in the deepfake condition in Study 2.

Study 1 pre-analysis plan, p. 4. Trust is measured on a 4-point scale (ranging from “0 = not at all confident” to “4 = completely confident”).

The problem

These are misprints in titles and notes. In Figures OA2a and OA2b the first two panels are headed Study 1 and the third is headed Study 2. The note to Figure OA2b repeats the definition of the three-point outcome of Figure OA2a, but Figure OA2b plots a binary outcome, as its title says. Its axis label “Dont Know if Real” is also inexact, since the question asks whether the politician truly believes the statement. The trust item has four response options, coded 0 to 3. Study 2 has no deepfake condition, and the pooled estimate in Figure OA3 excludes the deepfake arm of Study 1, as footnote 19 of the article says.

Evidence a check of the printed page

The evidence is a check of the pages against the survey instruments, the pre-analysis plans and the replication code. Both survey instruments list four options for the trust item, and both cleaning scripts code them 0 to 3. The code for Figure OA2b plots the indicator of the answer “I don’t know if the politician truly believes what is said in the video”, and the pooled regression for Figure OA3 selects the real-video participants of Study 1 and the Study 2 participants who knew Romney’s party.

What survives

The plotted estimates, the coding of the outcomes and the exclusion used in the pooled regression are unaffected, and only the titles, notes and axis label change. The Study 2 estimates in these figures were not recalculated.

Corrected

Figures OA2a and OA2b, titles (and the contents, p. 1). Figure OA2a: Studies 1 and 2 - Impacts of information about deepfakes on disbelief (3-point scale) […] Figure OA2b: Studies 1 and 2 - Impacts of information about deepfakes on disbelief (binary) […]

Figure OA2b, note. All three columns measure an indicator equal to 1 for “I don’t know if the politician truly believes what is said in the video,” and 0 for either alternative response. The label of the vertical axis becomes Don’t know if politician truly believes (binary).

Figure OA3, note. Responses range from 0 = “not at all confident” to 3 = “completely confident.” […] it excludes all participants in the deepfake condition in Study 1.

Study 1 pre-analysis plan, p. 4. Trust is measured on a 4-point scale (ranging from “0 = not at all confident” to “3 = completely confident”).

Crop of the title of Figure OA2a and the headings of its three panels. The title reads: Figure OA2a: Study 2 - Impacts of information about deepfakes on disbelief (3-point scale): unconvinced that politician believes what is being said. A red outline surrounds the words Study 2. The panel headings read: Impact of Information Above Deepfake Video (Study 1), Impact of Information Above Real Video (Study 1) and Impact of Information Above Real Video (Study 2). — Appendix, Figure OA2a, title and panel headings, p. 9. The outline marks “Study 2” in the title; the first two panels are headed Study 1.

Crop of the title of Figure OA2b and the headings of its three panels. The title reads: Figure OA2b: Study 2 - Impacts of information about deepfakes on disbelief (binary): don't know if politician truly believes what is being said. A red outline surrounds the words Study 2. The panel headings read: Impact of Information Above Deepfake Video (Study 1), Impact of Information Above Real Video (Study 1) and Impact of Information Above Real Video (Study 2). — Appendix, Figure OA2b, title and panel headings, p. 10. The outline marks “Study 2” in the title; the first two panels are headed Study 1.

Crop of the eight-line italic note under Figure OA2b. Red outlines surround, over three lines, the words: how convinced are you that the politician believes what is being said? (3-point scale from 1 equals not at all convinced to minus 1 equals very convinced). The sentence that contains them begins: All three columns measure impact in terms of the same outcome. The rest of the note describes the rows of the figure and says that the two leftmost graphs of treatment effects use Manski bounds and the rightmost uses standard errors from an OLS regression. — Appendix, note to Figure OA2b, p. 10. The outlines mark the definition of the three-point outcome, which belongs to Figure OA2a; it runs over three lines.

Crop of the six-line italic note under Figure OA3. The third line reads: range from 0 equals not at all confident to 4 equals completely confident, and a red outline surrounds the digit 4. The last sentence reads: The pooled estimate is estimated using a regression with shared covariates; it excludes all participants in the deepfake condition in Study 2, and a second red outline surrounds the final words Study 2. — Appendix, note to Figure OA3, p. 18. The outlines mark the upper end of the trust scale, printed as 4, and the study named for the excluded deepfake condition, printed as Study 2.

P034-E015 Minor

Misprints and wrong cross-references: five references in the Appendix point to the wrong table or section, the last sentence of Appendix Section 6 says “pro-choice” where “pro-life” is meant, and the note to Table OA2 prints $p<.0369$ for $p=.0369$ .

Article: p. 2 (PDF p. 2); Figure 1 caption and footnote 8, p. 4 (PDF p. 4); reference list, p. 14 (PDF p. 14). Appendix: contents, p. 1; footnote 1, p. 3; heading of Section 3, p. 6; note to Table OA2, p. 8; Table OA3, p. 11; notes to Tables OA5 and OA6, pp. 13–14; footnote 4, p. 18; Section 5, p. 19; Section 6, p. 20 (the same PDF pages of the Appendix)

As printed

Article p. 2: “A cursory search of the five most popular news websites in the U.S. YouGov (according to 2021) for the term …”
Article p. 4, caption: “Figure 1: Schreenshot of the video used in Study 1.”
Article p. 4, footnote 8: “27% reponded that he was Independent”
Appendix pp. 1 and 6, heading: “SECITON 3: Discernment of Deepfakes Without Deepfake Information”
Appendix p. 3, footnote 1: “Swaim, B. & Nussbaum, J. (2016). The Perfect Presidential Stump Speech. FiveThirtyEight. November.” Article p. 14, reference list: Swaim, Barton, and Jeff Nussbaum. 2016. “The Perfect Presidential Stump Speech” (October 3, 2016).
Appendix p. 8, note to Table OA2: “ANOVA f-statistic=2.84 ( $p\colorbox{#ffe58a}{$<$}.0369$ ).”
Appendix p. 11, Table OA3, row labels: “Subjective Evaluation of Character (e.g., trusworthy, traitor, etc.)” and “Demographic Fact (e.g., white, male, bussinessman)”
Appendix pp. 13 and 14, notes to Tables OA5 and OA6: “As mentioned in the PAP, this analysis excludes participants who are likely supporters of alternative medicine. This is defined as ALL participants EXCEPT those who BOTH oppose government mandates for vaccination AND support government subsidies for acupuncture procedures.”
Appendix p. 18, footnote 4: “… was used in Study 2 and produced very similar treatment effects. See Table OA5.”
Appendix p. 19: “As shown in Section 3, Tables 3-4, we saw no statistically significant differences.”
Appendix p. 19: “As seen in Section 3, Table OA9, we find that participants surveyed before the election have higher levels (0.14 points more) of belief that the video they watched is deepfaked than participants surveyed after the election ( $p<.05$ ).”
Appendix p. 19: “Our results (see Section 3, Tables 6-7) are consistent with this theory in terms of estimate directions but are not statistically significant.”
Appendix p. 20: “While we do include results illustrating the interaction of the information treatment and Republican self-identification (see Table OA8), we cannot directly address the question of motivated reasoning with this data for three reasons.”
Appendix p. 20: “And finally, it is not clear that all rank-and-file Republicans identify being pro-choice as a clearly Republican position; 31% of surveyed Republicans identified themselves as pro-choice in 2019.”

The problem

Each item is a slip in spelling, in a label or in a cross-reference, and the surrounding text shows what was meant. In item 6 the printed expression is false as written.

The source of the ranking is YouGov (2021). The name and the year have changed places with the words “according to”.
The word is “Screenshot”.
The word is “responded”.
The word is “SECTION”, as in the headings of the other sections.
The two entries date the same article differently. I did not check the date against the article.
From the twelve counts of Table OA2, the statistic is $F(3,1356)=2.837$ with $p=0.03694$ . This value rounds to $.0369$ and is larger than $.0369$ , and the printed inequality is false. The note prints the rounded $p$ -value as a bound where an equality is meant.
The words are “trustworthy” and “businessman”.
“This” has no clear antecedent. The clause that follows describes the participants who are kept, and the sentence before it speaks of those who are excluded. The program for the supplementary materials keeps every participant except those who both support subsidies for acupuncture and oppose vaccine mandates, as the clause says. The note can be read as defining the excluded group by that clause, which would reverse the sample.
The results for the alternative wording are in Table OA7 (p. 14), whose title and note quote that wording. Table OA5 reports favorability, vote choice and support for essential oils in Study 1.
Section 3 of the Appendix contains only Figure OA1. The secondary results for Study 1 are in Section 4, Tables OA5 and OA6 (pp. 13–14).
The estimate for the election, $-0.14$ , is in Section 4, Table OA11 (p. 16). Table OA9 reports the interaction with being pro-life.
The interactions with party and with the stance on abortion are in Section 4, Tables OA8 and OA9 (p. 15).
Section 6 concerns dismissing the video as a deepfake. The interaction of the warning with Republican identification for those outcomes is in Table OA10 (p. 16), and Table OA8 reports it for vote choice and favorability.
The clause after the semicolon says that 31% of Republicans call themselves pro-choice. That share is a reason to doubt that all Republicans regard the pro-life position as their party’s position. It is no reason to doubt this of the pro-choice position, which the article (p. 6) calls “an unusual stance for a Republican”.

Evidence a recalculation from the source

The evidence is a check of the pages against the contents of the Appendix, with a recalculation for item 6 and, for item 8, a reading of the program for the supplementary materials. For item 6, I computed the one-way analysis of variance from the counts printed in Table OA2. The participant’s party is scored 1 (Republican), 2 (Independent) and 3 (Democrat), as in the replication files, and the four guesses are the groups. A direct computation gives $F(3,1356)=2.836964$ and $p=0.036942$ . For item 8, the condition in that program excludes a participant exactly when the participant both supports subsidies for acupuncture and opposes vaccine mandates.

What survives

No estimate, interval or conclusion changes. The $p$ -value of Table OA2 remains below $.05$ . The analyses of Tables OA5 and OA6 use the sample that the notes intend. The Appendix still says, correctly, that the data cannot settle the question of motivated reasoning.

Corrected

“the five most popular news websites in the U.S. (according to YouGov 2021) for the term …”
“Figure 1: Screenshot of the video used in Study 1.”
“27% responded that he was Independent”
“SECTION 3: Discernment of Deepfakes Without Deepfake Information”, in the contents and in the heading
The two entries should give the same date. The reference list of the article gives October 3, 2016, and the Appendix should agree with it once the date has been checked against the article.
“ANOVA F-statistic=2.84 ( $p\colorbox{#ffe58a}{$=$}.0369$ ).”
“trustworthy” and “businessman”
“The retained analysis sample comprises all participants except those who both oppose government mandates for vaccination and support government subsidies for acupuncture procedures.”
“See Table OA7.”
“As shown in Section 4, Tables OA5–OA6, we saw no statistically significant differences.”
“As seen in Section 4, Table OA11, we find that participants surveyed before the election …”
“Our results (see Section 4, Tables OA8–OA9) are consistent with this theory …”
“(see Table OA10)”
“… identify being pro-life as a clearly Republican position; 31% of surveyed Republicans identified themselves as pro-choice in 2019.”

Crop of the lower edge of Figure 1, its caption and the first line of the next paragraph. A red box marks the word 'Schreenshot' in the caption 'Figure 1: Schreenshot of the video used in Study 1.' — Article p. 4 (PDF p. 4), caption of Figure 1, item 2. The marked word is misspelled.

Crop of five lines of the Introduction. A red box marks the words 'YouGov (according to 2021)' in the sentence 'A cursory search of the five most popular news websites in the U.S. YouGov (according to 2021) for the term deepfake on the news aggregator Google News returned 11,700 news articles discussing deepfakes'. — Article p. 2 (PDF p. 2), second paragraph, item 1. In the marked words the name of the source stands outside the parenthesis and the year inside it.

Crop of five footnote lines. A red box marks the word 'reponded' in the sentence '27 percent thought the politician was a Democrat, 16 percent thought he was a Republican, and 27 percent reponded that he was Independent.' — Article p. 4 (PDF p. 4), footnote 8, item 3. The marked word is misspelled.

Crop of the top of the page. A red box marks the word 'SECITON' in the heading 'SECITON 3: Discernment of Deepfakes Without Deepfake Information'. Two lines of text follow. — Appendix p. 6, heading of Section 3, item 4. The marked word is misspelled; the contents on p. 1 of the Appendix have the same spelling.

Crop of a footnote: '1 See Swaim, B. \& Nussbaum, J. (2016). The Perfect Presidential Stump Speech. FiveThirtyEight. November. Retrieved from' followed by a link. A red box marks the word 'November.' — Appendix p. 3, footnote 1, item 5. The marked month is the date given for the FiveThirtyEight article.

Crop of the reference entry 'Swaim, Barton, and Jeff Nussbaum. 2016. The Perfect Presidential Stump Speech (October 3, 2016).' followed by a link, and of the first line of the next entry. A red box marks '(October 3, 2016)'. — Article p. 14 (PDF p. 14), reference list, item 5. The marked date is given for the same article.

Crop of the last two rows of Table OA2 (totals 369, 406, 370, 215 and 1,360) and the note under the table: 'Lucid asks for participants' political party before they are able to enroll in any surveys. ANOVA f-statistic=2.84 (p less than .0369).' A red box marks the parenthesis with the p-value. — Appendix p. 8, note to Table OA2, item 6. The marked expression gives the rounded $p$ -value as an upper bound.

Crop of six rows of Table OA3 with their percentages and counts. Red boxes mark the word 'trusworthy' in the label 'Subjective Evaluation of Character (e.g., trusworthy, traitor, etc.)' and the word 'bussinessman' in the label 'Demographic Fact (e.g., white, male, bussinessman)'. — Appendix p. 11, Table OA3, item 7. The two marked words in the row labels are misspelled.

Crop of the last row of Table OA5 (95 percent lower bound: -0.73, -0.31, -0.75) and the note under it. Red boxes mark the words 'This is defined as ALL participants EXCEPT', split over two lines, in the note 'As mentioned in the PAP, this analysis excludes participants who are likely supporters of alternative medicine. This is defined as ALL participants EXCEPT those who BOTH oppose government mandates for vaccination AND support government subsidies for acupuncture procedures.' — Appendix p. 13, note to Table OA5, item 8; the note to Table OA6 on p. 14 is the same. After the marked words the note describes the participants who are kept, and the sentence before them speaks of those who are excluded.

Crop of footnote 4 at the foot of the page, with the page number 18. A red box marks the words 'See Table OA5.' at the end of the footnote, after 'was used in Study 2 and produced very similar treatment effects.' — Appendix p. 18, footnote 4, item 9. The marked reference is to Table OA5; the results for the alternative wording are in Table OA7.

Crop of three lines. A red box marks the words 'Section 3, Tables 3-4,' in the sentence 'As shown in Section 3, Tables 3-4, we saw no statistically significant differences.' — Appendix p. 19, Section 5, Study 1, item 10. The marked reference names Section 3 and Tables 3-4; the results are in Section 4, Tables OA5 and OA6.

Crop of five lines. A red box marks the words 'Section 3, Table OA9,' in the sentence 'As seen in Section 3, Table OA9, we find that participants surveyed before the election have higher levels (0.14 points more) of belief that the video they watched is deepfaked than participants surveyed after the election'. — Appendix p. 19, Section 5, Study 2, item 11. The marked reference names Section 3 and Table OA9; the estimate of 0.14 points is in Section 4, Table OA11.

Crop of three lines. A red box marks the words 'Section 3, Tables 6-7)' in the sentence 'Our results (see Section 3, Tables 6-7) are consistent with this theory in terms of estimate directions but are not statistically significant.' — Appendix p. 19, Section 5, Pooled, item 12. The marked reference names Section 3 and Tables 6-7; the interactions are in Section 4, Tables OA8 and OA9.

Crop of the last eight lines of Section 6. Red boxes mark 'Table OA8)' in 'the interaction of the information treatment and Republican self-identification (see Table OA8)' and the first 'pro-choice' in 'it is not clear that all rank-and-file Republicans identify being pro-choice as a clearly Republican position; 31 percent of surveyed Republicans identified themselves as pro-choice in 2019.' — Appendix p. 20, Section 6, items 13 and 14. The marked reference is to Table OA8, and the marked “pro-choice” is followed by the share of Republicans who call themselves pro-choice.

P034-E018 Minor

The Harwell reference should be dated May 14, 2021, not March 14, and the reference “Ternovski and Orr 2022” joins the authors and year of a 2022 journal article to the title and link of a 2020 memo by four authors.

References, entry “Harwell, Drew. 2021.”, p. 13 (PDF p. 13), and entry “Ternovski, John, and Lilla Orr. 2022.”, p. 14 (PDF p. 14), which is cited in footnote 6, p. 3 (PDF p. 3)

As printed

References, p. 13. Harwell, Drew. 2021. “Remember the `deepfake cheerleader mom'? Prosecutors now admit they can’t prove fake-video claims” (March 14, 2021). https://www.washingtonpost.com/technology/2021/05/14/deepfake-cheer-mom-claims-dropped/.

References, p. 14. Ternovski, John, and Lilla Orr. 2022. “Evidence of rising rates of inattentiveness on Lucid in 2020,” https://osf.io/preprints/socarxiv/8sbe4/.

Footnote 6, p. 3. During the COVID-19 pandemic, Ternovski and Orr (2022) found that Lucid data can provide reliable data when researchers screen on attentiveness, which we do here.

The problem

The reference list has two slips. In the Harwell entry, the date in parentheses is March 14, 2021, while the publisher dates the article May 14, 2021, as the link of the entry itself does (2021/05/14).

The second entry joins the data of two publications. The title “Evidence of rising rates of inattentiveness on Lucid in 2020” and the SocArXiv identifier 8sbe4 belong to a memo that is catalogued under four authors, Aronow, Kalla, Orr and Ternovski (2020). The pre-analysis plan of Study 2 cites the memo under those four names (p. 6, footnote 3). The authors and year of the entry, Ternovski and Orr (2022), belong to the journal article “A Note on Increases in Inattentive Online Survey-Takers Since 2020”, which has a different title and a different DOI. The entry does not identify which of the two footnote 6 relies on, and the evidence does not determine which one the entry was meant to cite.

Evidence a check of the printed page

The evidence is a check of the pages against the public pages of the cited sources. The page of the Harwell article at the publisher displays the date May 14, 2021 below the headline. The journal lists Ternovski and Orr as the authors of “A Note on Increases in Inattentive Online Survey-Takers Since 2020” (Journal of Quantitative Description: Digital Media 2, DOI 10.51685/jqd.2022.002, published on 7 February 2022). A library catalogue entry for the memo, with DOI 10.31235/osf.io/8sbe4, lists the four authors and the date 15 September 2020. The full text of the memo and the history of its versions on OSF were not obtained. These sources do not exclude that the two works are stages of one project or that the author list of the memo changed between versions.

What survives

No result of the paper depends on either entry. I did not establish whether the statement of footnote 6 about Lucid is correct. Both works favour screening on attentiveness. The journal article adds that attention checks “do contain drawbacks and are not a panacea” (pp. 25–26), and the abstract of the memo recommends stringent attention checks.

Corrected

References, p. 13. Harwell, Drew. 2021. “Remember the `deepfake cheerleader mom'? Prosecutors now admit they can’t prove fake-video claims” (May 14, 2021). https://www.washingtonpost.com/technology/2021/05/14/deepfake-cheer-mom-claims-dropped/.

References, p. 14. The entry should cite one work in full. If the memo is the source of footnote 6, then footnote 6 cites Aronow et al. (2020) and the entry is

Aronow, P. M., Joshua Kalla, Lilla Orr, and John Ternovski. 2020. “Evidence of rising rates of inattentiveness on Lucid in 2020,” https://osf.io/preprints/socarxiv/8sbe4/.

If the journal article is the source, then the entry is

Ternovski, John, and Lilla Orr. 2022. “A Note on Increases in Inattentive Online Survey-Takers Since 2020.” Journal of Quantitative Description: Digital Media 2. https://doi.org/10.51685/jqd.2022.002.

In the second case, footnote 6 should describe screening on attentiveness as a way to reduce measurement error, to be used with continuing checks of data quality, and not as sufficient to validate the data.

Crop of three entries of the reference list. The middle entry reads: Harwell, Drew. 2021. Remember the 'deepfake cheerleader mom'? Prosecutors now admit they can't prove fake-video claims (March 14, 2021). https://www.washingtonpost.com/technology/2021/05/14/deepfake-cheer-mom-claims-dropped/. A red outline surrounds the word March. The entry above is H.R.6395, the National Defense Authorization Act for Fiscal Year 2021, and the entry below is Kuklinski et al. 2000. — References, p. 13 (PDF p. 13): the Harwell entry between its two neighbours. The outline marks the month “March”; the link printed in the same entry carries `2021/05/14`.

Crop of three entries of the reference list. The middle entry reads: Ternovski, John, and Lilla Orr. 2022. Evidence of rising rates of inattentiveness on Lucid in 2020, https://osf.io/preprints/socarxiv/8sbe4/. A red outline surrounds the words Ternovski, John, and Lilla Orr. 2022. The entry above is Swire et al. 2017 and the entry below is Thalen 2021, which is dated May 17, 2021. — References, p. 14 (PDF p. 14): the entry cited in footnote 6, between its two neighbours. The outline marks the authors and year, which are those of the 2022 journal article; the title and link that follow are those of the 2020 memo.

Crop of footnote 6. It reads: Lucid is an increasingly popular alternative to Amazon Mechanical Turk for social science survey research. Many well-known findings have been replicated on Lucid, suggesting the platform is capable of providing high-quality data (Coppock and McClellan 2019; Peyton, Huber, and Coppock 2021). During the COVID-19 pandemic, Ternovski and Orr (2022) found that Lucid data can provide reliable data when researchers screen on attentiveness, which we do here. A red outline surrounds the words Ternovski and Orr (2022). — Footnote 6, p. 3 (PDF p. 3). The outline marks the citation “Ternovski and Orr (2022)”, which the footnote gives as the source of its statement about screening on attentiveness.

P034-E019 Minor

In the replication files, the device indicator of Study 2 is reversed, the supplementary script uses the covariate macro before defining it when only tables are requested, and the saved tables for Tables OA7, OA11 and OA12 differ from the printed ones; no printed estimate is shown to be wrong.

Section 3.1, p. 9 (PDF p. 9); Appendix, Tables OA7 (p. 14), OA11 (p. 16) and OA12 (p. 17), and the saved tables for them; replication code, lines 313–315 of 2-study2_data_cleanup_FINAL.do and lines 6–7, 593–766 and 840–841 of 4-Supplementary-Materials_FINAL.do

As printed

2-study2_data_cleanup_FINAL.do, lines 313–315.

/*respondent's device: computer fl_25 ; mobile fl_87*/

gen mobile=0 if fl_25=="" & fl_87!=""

replace mobile=1 if fl_25!="" & fl_87==""

4-Supplementary-Materials_FINAL.do, lines 6, 7 and 594.

local figures=0

local tables=0

reg trust_tv_actually tr_warn $covars if romney_party==1 & study==2, robust

Appendix, Table OA11, p. 16, column “Deepfake?”. Post Election Indicator: $-0.14$ . 95% Upper Bound: $-0.01$ . 95% Lower Bound: $-0.27$ .

Section 3.1, p. 9. We found some evidence that participants surveyed before the election displayed higher levels of distrust in political videos than those surveyed afterwards ( $p<0.05$ ).

The problem

There are three defects in the replication files. First, the cleaning script of Study 2 assigns mobile the values opposite to its own comment and to the cleaning script of Study 1: a participant with only the mobile route fl_87 filled gets $0$ , and a participant with only the computer route fl_25 filled gets $1$ . Second, the supplementary script has one switch for figures and one for tables, and every definition of the global macro covars before line 840 lies inside a block that runs only when the switch for figures is on. With the switch for tables on and the switch for figures off, 16 regressions in the table blocks read the macro before it is defined, among them the regressions for Tables OA7, OA11 and OA12. The macro is then empty in a fresh Stata session, and it holds the six covariates of Study 1 if the main analysis script has run first, as the README instructs; in neither case do these regressions use the expanded covariate list of Study 2. Third, the saved tables for Tables OA7, OA11 and OA12 differ from the printed tables in 5 of 6, 4 of 9 and 3 of 9 entries. In the first column of Table OA11, the saved estimate is $-0.121$ with the interval $[-0.255, 0.012]$ , which contains zero. The printed entry is $-0.14$ with the interval $[-0.27, -0.01]$ , and the sentence on p. 9 reports $p<0.05$ . The replication files do not show which specification or data produced the saved tables, or whether the undefined macro explains the difference.

Evidence a recalculation from the source

I checked the two device assignments, traced the supplementary script and compared the saved tables with the printed ones. I then recalculated Tables OA7, OA11 and OA12 from the Study 2 data in P034-N011, translating the relevant code and using the supplied cleaned covariates. Stata was not run. The recalculation reproduces the printed tables under the code sample; it does not establish how the discrepant saved workbooks were produced.

The columns of Tables OA7, OA11 and OA12 in which the saved tables differ from the printed ones. Each entry is an estimate with its lower and upper bounds, and the saved values are rounded to three decimals.
Table	Column	Saved in the replication files	Printed
OA7	Trust TV	$-0.012$ $[-0.095, 0.070]$	$0.01$ $[-0.07, 0.09]$
OA7	Trust Online	$-0.032$ $[-0.117, 0.052]$	$-0.04$ $[-0.12, 0.04]$
OA11	Deepfake?	$-0.121$ $[-0.255, 0.012]$	$-0.14$ $[-0.27, -0.01]$
OA11	Don’t Know if Real	$-0.003$ $[-0.047, 0.042]$	$-0.01$ $[-0.05, 0.04]$
OA12	Deepfake?	$0.132$ $[-0.135, 0.400]$	$0.12$ $[-0.14, 0.39]$

In the first column of Table OA11, the width of the saved interval implies a standard error of $0.0681$ . The saved estimate is then $1.777$ standard errors from zero, which is less than $1.96$ . The saved tables for Tables OA5, OA6, OA8, OA9 and OA10 agree with the printed tables in every entry.

What survives

No printed estimate, interval or $p$ -value is shown to be wrong. The device indicator enters no regression. In Tables OA7 and OA12, every interval contains zero in the saved version and in the printed one. The main analysis script and the supplementary script each define the covariates of Study 2 before their own regression for the comparison across the election, so the model behind the sentence on p. 9 is specified apart from the table blocks. In the recalculation of P034-N011, I reproduce all 24 printed entries of Tables OA7, OA11 and OA12 with the code sample; the election comparison has $p=0.038$ , and $p=0.047$ with the plan sample, so the $p<0.05$ sentence stands. This does not establish how the discrepant saved tables were produced.

Corrected

2-study2_data_cleanup_FINAL.do, lines 314–315.

gen mobile=1 if fl_25=="" & fl_87!=""

replace mobile=0 if fl_25!="" & fl_87==""

4-Supplementary-Materials_FINAL.do. The definition of lines 840–841 is to be placed before line 593, outside the blocks for figures:

global covars c.age i.gender i.party i.region c.hhi elder-pro_vax conlib ///

imputed_conlib bigfive* pol_know_total polinterest

I reproduce the printed tables in the translated calculation of P034-N011. After the native scripts are repaired, the saved workbooks for Tables OA7, OA11 and OA12 should be regenerated from the Study 2 data and replaced with verified outputs, with any remaining difference from the printed tables explained.

Crop of six lines of the article under the heading on heterogeneous treatment effects. The marked sentence reads: We found some evidence that participants surveyed before the election displayed higher levels of distrust in political videos than those surveyed afterwards (p less than 0.05). A red outline surrounds the parenthetical p less than 0.05. — Section 3.1, p. 9 (PDF p. 9). The outline marks the $p$ -value reported for the difference between participants surveyed before and after the election.

Crop of Table OA11 with its title: Study 2, Some evidence of more distrust immediately before a major election. The table has columns Deepfake?, Unconvinced and Don't Know if Real. Red outlines surround the three entries of the first column: minus 0.14 for the Post Election Indicator, minus 0.01 for the 95 percent Upper Bound and minus 0.27 for the 95 percent Lower Bound. The other columns read minus 0.05, 0.02, minus 0.12 and minus 0.01, 0.04, minus 0.05. — Appendix, Table OA11, p. 16. The outlines mark the three entries of the column “Deepfake?”, which differ from the table saved in the replication files.

Listing of four numbered lines of Stata code under the heading ORIGINAL CODE: 2-study2_data_cleanup_FINAL.do, SOURCE LINES 312-315. Line 313 is the comment: respondent's device: computer fl_25 ; mobile fl_87. Line 314 generates mobile equal to 0 if fl_25 is empty and fl_87 is not empty. Line 315 replaces mobile by 1 if fl_25 is not empty and fl_87 is empty. Nothing is outlined. — Replication code, `2-study2_data_cleanup_FINAL.do`, lines 312–315. The comment on line 313 gives the mapping of the two routes to devices, and lines 314–315 assign `mobile` the opposite values.

Listing of numbered excerpts of Stata code under the heading ORIGINAL CODE: 4-Supplementary-Materials_FINAL.do. Lines 3 to 7 include local figures=0 and local tables=0. Lines 190 to 193 open a block with the condition that figures equals 1. Lines 207 to 211 define the global macro covars as c.age i.gender i.party i.region c.hhi elder-pro_vax conlib imputed_conlib bigfive* pol_know_total polinterest and run a regression with it. Lines 597 to 604 set up the saved table trust_SM_table.xlsx and, on line 603, regress trust_tv_actually on tr_warn and the macro covars for Study 2 participants with romney_party equal to 1. Nothing is outlined. — Replication code, `4-Supplementary-Materials_FINAL.do`, excerpts. Lines 6–7 set both switches to zero, lines 207–208 define `covars` inside the block for figures that opens at line 190, and line 603 is a table regression that reads the macro.

P034-E003 Minor

The $p<0.001$ printed for the Study 1 effect in the deepfake-video arm is the complete-case value; in the worst-case regression that the Appendix prescribes for statistical significance, $p \approx 0.003$ , and the pooled $p$ -values are computed without the bounds.

Section 3, paragraph beginning “The leftmost panel of Figure 2”, p. 7 (PDF p. 7); Appendix, Section 2, Study 1, p. 5; pooled estimates on pp. 8–9 and in the Appendix, Section 5, p. 19

As printed

Section 3, p. 7. The leftmost panel of Figure 2 on the next page illustrates that when the information about deepfakes was randomly added to a deepfake video, participants were 0.5 points more likely to believe the video they were watching was fabricated ( $\colorbox{#ffe58a}{$p<0.001$}$ ).

Appendix, Section 2, Study 1, p. 5. The major deviation is the use of Manski bounds for confidence intervals and statistical significance. […] We impose the distribution of outcomes that least favors our hypothesis onto all missing outcomes (i.e., the survey instrument maximum for all missing outcomes in the control and the survey instrument minimum for all missing outcomes in treatment), run a covariate-adjusted OLS regression with robust standard errors over all outcomes, and use the lower bound of the resulting 95% confidence interval as the lower bound of our reported 95% CI.

The problem

The Appendix states that, in Study 1, bounds are used for confidence intervals and for statistical significance. To form the lower end of an interval, each missing outcome is set to the value that least favors the hypothesis, and a covariate-adjusted regression with robust standard errors is fitted to all randomized participants. I call this the worst-case regression, after the “worst-case bounds” of p. 5 of the article, and I call the regression with the opposite values the best-case regression. The $p<0.001$ on p. 7 does not come from this procedure. It is the $p$ -value of the complete-case regression, which drops the 15 participants in the deepfake-video arm whose outcome is missing. In the worst-case regression the two-sided $p$ -value is $0.0030$ . Under the method that the Appendix announces, the effect is therefore significant at the 1% level and not at the 0.1% level. The page does not say which regression the parenthetical reports. The program that computes the bounds returns a point estimate and the two ends of the interval, and no $p$ -value.

The pooled estimates raise a second point. The estimates of 0.40 points, 0.13 points and 6.0 percentage points on pp. 8–9, and of 3.8 percentage points and 0.17 points in Section 5 of the Appendix, come from robust regressions with a study fixed effect and shared covariates. These regressions include the real-video arm of Study 1, where 17 control participants and 4 treated participants did not complete the survey, and they do not use the Study 1 bounds. Four of the five regressions drop participants whose outcome is missing. The fifth, for intention to vote, uses an indicator that the vote equals yes, which codes the 11 missing votes in the real-video arm of Study 1 as zero (see P034-E011). The paper does not promise bounds for the pooled analyses, and the note to Figure OA3 separates the Study 1 bounds from the pooled regression for the outcomes of that figure. The pooled $p$ -values are not shown to be wrong, but the text does not say that they are computed without the bounds.

Evidence a recalculation from the source

The evidence is a recalculation from the Study 1 raw data, following the cleaning and analysis code in the replication files, together with an exact lower bound for the $p$ -value of the worst-case regression. The deepfake-video arm has 349 control participants, of whom 6 have a missing outcome, and 351 treated participants, of whom 9 have a missing outcome. The worst-case regression keeps every observed outcome, sets the 6 missing control outcomes to $3$ and sets the 9 missing treated outcomes to $-3$ ; these are the ends of the scale. Each regression has an intercept, the treatment indicator and seven covariate columns, and its standard errors are the robust ones that Stata reports.

The effect of the information treatment in the deepfake-video arm of Study 1 in the three regressions.
Regression	$n$	Estimate	Robust SE	Degrees of freedom	Two-sided $p$
Complete case	685	0.4988516	0.1163098	676	0.0000206
Worst case	700	0.3549310	0.1193355	691	0.0030396
Best case	700	0.6235811	0.1178226	691	0.0000002

Following the rule of the Appendix, with the multiplier $1.96$ that the program uses, I obtain the ends of the 95% interval:

0.3549310 - 1.96 \times 0.1193355 \approx 0.121033, \qquad 0.6235811 + 1.96 \times 0.1178226 \approx 0.854513 .

This interval agrees with the one drawn in the left panel of Figure 2, and it excludes zero.

In the worst-case regression, the $t$ statistic is $2.974$ on 691 degrees of freedom. The bound $p > 0.001$ does not depend on a numerical evaluation of the $t$ distribution. In exact rational arithmetic, the estimate $b$ and its robust variance $\widehat{V}$ satisfy $b > 0$ and $b^2/\widehat{V} < 9$ (the ratio is $8.846$ ), which implies that $0 < t < 3$ . The density of the $t$ distribution with 691 degrees of freedom is

f(x) = \frac{C}{\pi \sqrt{691}} \Bigl( 1 + \frac{x^2}{691} \Bigr)^{-346}, \qquad C = \frac{4^{345} (345!)^2}{690!},

and it is decreasing for $x > 0$ . Since $t < 3$ , I have

p = 2 \int_t^\infty f(x) \, dx > 2 \int_3^{33/10} f(x) \, dx \ge \frac{3}{5} f\Bigl( \frac{33}{10} \Bigr) > \frac{3}{5} \cdot \frac{C}{(22/7) \cdot 27} \Bigl( 1 + \frac{(33/10)^2}{691} \Bigr)^{-346} .

In the last inequality, I used that $\pi < 22/7$ and $\sqrt{691} < 27$ . The right side is a rational number, approximately $0.0010409$ , and it exceeds $1/1000$ . A two-sided normal test gives $0.0029$ and a one-sided $t$ test gives $0.0015$ ; neither restores the printed threshold. One admissible completion of the missing outcomes suffices for this conclusion; I do not claim that $0.0030$ is the largest $p$ -value over all completions.

In the real-video arm of Study 1, I find $p = 0.0000165$ in the worst-case regression, with the interval $[0.2877, 1.0999]$ . The code shows that the pooled regressions omit the bounds. I reproduced the three pooled estimates of pp. 8–9 without bounds, as reported in Reproduction. I did not compute the pooled regressions with the Study 1 bounds.

What survives

The complete-case estimate of 0.5 points, the intervals in Figure 2 and the positive effect in the deepfake-video arm at the 5% level are unaffected, and the effect remains significant at the 1% level in the worst-case regression. In the real-video arm of Study 1, $p<0.001$ holds in the worst-case regression as well. The Study 2 analyses do not use the bounds. No pooled estimate or pooled $p$ -value is shown to be wrong; whether the pooled results remain significant with bounds was not checked. What changes is the significance level on p. 7 under the method of the Appendix, and the description of the pooled analyses.

Corrected

Section 3, p. 7. […] participants were 0.5 points more likely to believe the video they were watching was fabricated (complete-case regression, $n = 685$ : $p<0.001$ ; with missing outcomes set to the values that least favor the hypothesis, as in Section 2 of the Appendix, $n = 700$ : estimate $0.35$ , $p \approx 0.003$ ; the 95% interval formed from the worst- and best-case regressions is $[0.12, 0.85]$ ).

Section 3, pp. 8–9, and Appendix, Section 5, p. 19. The pooled estimates and their $p$ -values come from robust regressions with a study fixed effect and shared covariates, without the Study 1 bounds; the pooled regression for intention to vote codes a missing vote as zero.

Crop of six lines of the article. The text reads: The leftmost panel of Figure 2 on the next page illustrates that when the information about deepfakes was randomly added to a deepfake video, participants were 0.5 points more likely to believe the video they were watching was fabricated (p less than 0.001), followed by footnote mark 14 and the start of the next sentences. A red outline surrounds the parenthetical p less than 0.001. — Section 3, p. 7 (PDF p. 7). The outline marks the $p$ -value printed for the effect of 0.5 points in the deepfake-video arm of Study 1.

Crop of the eight-line paragraph of the Appendix that begins: The major deviation is the use of Manski bounds for confidence intervals and statistical significance. A red outline surrounds the words for confidence intervals and statistical significance. The paragraph goes on to say that the distribution of outcomes that least favors the hypothesis is imposed on all missing outcomes, the survey instrument maximum in the control and the minimum in treatment, that a covariate-adjusted regression with robust standard errors is run, and that the lower bound of its 95 percent confidence interval is the lower bound of the reported interval, with the same done for a best-case distribution to obtain the upper bound. — Appendix, Section 2, Study 1, p. 5. The outline marks the statement that the bounds are used for confidence intervals and statistical significance; the rest of the paragraph describes how missing outcomes are filled in.

P034-E005 Minor

The bounding program replaces every outcome of the 36 respondents who did not answer the last survey item, including answers that they gave, and ten intervals for secondary outcomes of Study 1 are wider than under the procedure described in the Appendix.

Appendix, Section 2, Study 1, p. 5 (PDF p. 5 of the Appendix), sentence beginning “We impose the distribution of outcomes”; replication files, manski_ci.ado, lines 19–21 and 36–38. The article refers to this section on p. 5 (PDF p. 5)

As printed

Appendix p. 5: “We impose the distribution of outcomes that least favors our hypothesis onto all missing outcomes (i.e., the survey instrument maximum for all missing outcomes in the control and the survey instrument minimum for all missing outcomes in treatment), run a covariate-adjusted OLS regression with robust standard errors over all outcomes, and use the lower bound of the resulting 95% confidence interval as the lower bound of our reported 95% CI. We do the same with a best-case distribution of missing outcomes to estimate the upper bound of our reported 95% CI.”

manski_ci.ado, lines 19–21 (the worst case):

gen `outcomenew'=`outcome'

replace `outcomenew'=`max' if `treat'==0 & `complete'==0

replace `outcomenew'=`min' if `treat'==1 & `complete'==0

Lines 36–38 (the best case) are the same with max and min exchanged.

The problem

The Appendix describes the bounds outcome by outcome: the extreme values are imposed on the missing values of the outcome that is analyzed, and the observed answers are kept. The program does something else. I write $Y_i$ for the outcome analyzed, $T_i$ for the treatment indicator passed to the program, and $M_i$ for the indicator that $Y_i$ is missing. Let $C_i$ denote the completion flag, which in every call is the indicator that respondent $i$ answered the last item of the survey, the guess of the politician’s party. The program replaces $Y_i$ by an extreme value when $C_i=0$ , and not when $M_i=1$ . A respondent who answered an earlier question and then left the survey has $C_i=0$ and $M_i=0$ for that question, and the answer is overwritten in at least one of the two fits. The two rules agree for an outcome only if $M_i=1$ exactly when $C_i=0$ .

In Study 1, 36 of the 1,396 randomized respondents did not answer the last item. All 36 are missing on the main outcome, the belief that the video is a deepfake, and on the question about essential oils, and for these two outcomes the rules coincide. For the other outcomes they do not. Of the 36, 11 answered the questions on favorability and on the vote, 6 answered the two questions on whether the politician believes what is said, and 2 answered each of the two questions on trust.

Evidence an exact counterexample

The evidence is one respondent in the data who shows the discrepancy exactly, and a recalculation from the Study 1 raw data. A respondent in an arm without the warning did not answer the last item and answered “Extremely unfavorable” ( $-3$ ) to the question on favorability. In the fit for the lower endpoint, the program replaces this answer by $+3$ . The rule of the Appendix keeps $-3$ in both fits.

I rebuilt the Study 1 analysis file from the raw data with the released cleaning rules. I then computed each interval twice, once as the program does and once replacing only missing values. The first computation agrees with all 18 printed entries of Tables OA5 and OA6. Of the 17 intervals computed, ten change.

The ten intervals that change, as the program computes them and when only missing values are replaced. The Appendix prints the first four to two decimals and shows the other six in figures.
Interval	As coded	Only missing values replaced
Table OA5, favorability	$[-0.732,\,0.248]$	$[-0.611,\,0.167]$
Table OA5, vote	$[-0.311,\,0.079]$	$[-0.265,\,0.058]$
Table OA6, favorability	$[-0.322,\,0.669]$	$[-0.217,\,0.595]$
Table OA6, vote	$[-0.162,\,0.235]$	$[-0.119,\,0.220]$
Figure OA1, unconvinced	$[-0.169,\,0.187]$	$[-0.156,\,0.176]$
Figure OA1, don’t know	$[-0.123,\,0.085]$	$[-0.117,\,0.079]$
Figure OA2a, real video	$[0.014,\,0.358]$	$[0.033,\,0.340]$
Figure OA2b, real video	$[-0.045,\,0.157]$	$[-0.036,\,0.148]$
Figure OA3, trust in footage on TV	$[-0.094,\,0.233]$	$[-0.087,\,0.230]$
Figure OA3, trust in footage online	$[-0.221,\,0.103]$	$[-0.216,\,0.098]$

Each interval as coded contains the interval obtained when only missing values are replaced. The seven intervals that do not change are the two of Figure 2, the first column of Figure OA1, the panels for the deepfake video in Figures OA2a and OA2b, and the columns for essential oils in Tables OA5 and OA6. No point estimate changes, and each of the 17 intervals contains zero under one rule if and only if it does under the other.

What survives

The main result is untouched: for the outcome of Figure 2 the two rules replace the same 36 values, and the fits are identical. All point estimates stand. The conclusions drawn from the ten intervals also stand, since none changes its position relative to zero. In particular, the interval for the real video in Figure OA2a stays above zero. The Study 2 regressions and the pooled regressions do not use the program. What changes is the width of ten intervals for secondary outcomes of Study 1. That the wider intervals contain the narrower ones in these data is not a statement about coverage, and no failure of coverage is established.

Corrected

Either the description or the program can be changed. To keep the printed numbers, the Appendix should say what the program does: “For every outcome, we treat all answers of a respondent who did not answer the final survey item as missing, including answers given to earlier questions. We impose the distribution of outcomes that least favors our hypothesis onto all missing outcomes …”

To carry out the procedure as described, each of the four replacement conditions (lines 20, 21, 37 and 38 of manski_ci.ado) should test the outcome itself. For example, line 20 becomes

replace `outcomenew'=`max' if `treat'==0 & missing(`outcome')

The Appendix can then read: “For each outcome, we retain every observed answer and replace only missing values with the instrument maximum in control and minimum in treatment for the lower endpoint, reversing these assignments for the upper endpoint. We fit the same adjusted regressions with robust standard errors and recompute the affected intervals.”

In full, let $[L,U]$ be the range of the instrument. The two imputed outcomes are

Y_i^{\mathrm{low}}=\begin{cases}Y_i&\text{if }M_i=0,\\U&\text{if }M_i=1\text{ and }T_i=0,\\L&\text{if }M_i=1\text{ and }T_i=1,\end{cases} \qquad Y_i^{\mathrm{up}}=\begin{cases}Y_i&\text{if }M_i=0,\\L&\text{if }M_i=1\text{ and }T_i=0,\\U&\text{if }M_i=1\text{ and }T_i=1.\end{cases}

The reported interval is $[\hat\beta^{\mathrm{low}}-1.96\,\widehat{\mathrm{se}}^{\mathrm{low}},\ \hat\beta^{\mathrm{up}}+1.96\,\widehat{\mathrm{se}}^{\mathrm{up}}]$ , where $\hat\beta$ and $\widehat{\mathrm{se}}$ are the coefficient on $T_i$ and its robust standard error in the covariate-adjusted regression of the imputed outcome. With this rule the intervals of Table OA5 become, at the printed precision, $[-0.61,0.17]$ for favorability and $[-0.26,0.06]$ for the vote, and those of Table OA6 become $[-0.22,0.59]$ and $[-0.12,0.22]$ . The intervals for essential oils are unchanged.

Crop of six lines of Section 2 of the Appendix. Two red boxes mark the words 'onto all missing outcomes', split over two lines, in the sentence 'We impose the distribution of outcomes that least favors our hypothesis onto all missing outcomes (i.e., the survey instrument maximum for all missing outcomes in the control and the survey instrument minimum for all missing outcomes in treatment)'. — Appendix, Section 2, p. 5 (PDF p. 5 of the Appendix). The marked words say which values are replaced by the extremes: the missing outcomes.

Rendering of fourteen numbered lines of Stata code under the headings 'ORIGINAL CODE: manski_ci.ado' and 'SOURCE LINES 14-27'. Red boxes mark the condition complete==0, written with Stata macro quotes, at the end of line 20, 'replace outcomenew=max if treat==0 and complete==0', and of line 21, 'replace outcomenew=min if treat==1 and complete==0'. Each of the two lines carries the comment 'replace missing outcomes with bounds'. — Replication files, `manski_ci.ado`, lines 14–27, in a rendering of the released file. The two marked conditions select every respondent whose completion flag is zero, whether or not the outcome analyzed is missing.

P034-A001 Minor

Table OA5 reports the effect of the information treatment for the real video, and Table OA6 the effect of the deepfake without the information treatment; these are the main coefficients of models that contain the interaction of the two treatments, and not effects averaged over the other treatment.

Appendix, Table OA5, p. 13, and Table OA6, p. 14 (titles, row labels and notes); cited from Section 3.1, p. 9 (PDF p. 9)

See also under Formal statements (Lean): P034-T3, P034-T4

As printed

Appendix, p. 13. Table OA5: Study 1 – Information about deepfakes has no impact on favorability, vote choice, or support for essential oils in medicine (but we are underpowered). Information Treatment Effect: $-0.22$ (Favor), $-0.10$ (Vote), $-0.25$ (More Essential Oils in Medicine?). 95% Upper Bound: $0.25$ , $0.08$ , $0.26$ . 95% Lower Bound: $-0.73$ , $-0.31$ , $-0.75$ .

Appendix, p. 14. Table OA6: Study 1 – Deepfakes have no impact on favorability, vote choice, or support for essential oils in medicine (but we are underpowered). Deepfake Treatment Effect: $0.19$ (Favor), $0.05$ (Vote), $-0.14$ (More Essential Oils in Medicine?). 95% Upper Bound: $0.67$ , $0.24$ , $0.39$ . 95% Lower Bound: $-0.32$ , $-0.16$ , $-0.63$ .

The problem

The row labels read as the effects of the two treatments. Neither the titles nor the notes say at which level of the other treatment each effect is taken. The models behind the tables contain both treatments and their interaction, and each table reports a main coefficient. Table OA5 therefore reports the effect of the information treatment for participants who saw the real video, and Table OA6 reports the effect of the deepfake for participants who did not receive the information treatment. An effect averaged over the other treatment is a different contrast, since it adds a share of the interaction coefficient to the main coefficient. For favorability, the $-0.22$ of Table OA5 can be taken for the averaged effect, which is $-0.295$ with equal weights. The secondary hypotheses of the Study 1 pre-analysis plan compare information with no information, and the deepfake with the real video, without conditioning on the other treatment; the tables do not report those contrasts. All 18 printed entries are correct for the contrasts that were fitted. The meaning of “no impact” is clear from the parenthetical in each title and from Section 5 of the Appendix: no difference is statistically significant with the bounds.

Evidence a historical Lean attribution (Generated status: clause support not certified in this release)

The evidence is a derivation of the two contrasts and a recalculation from the Study 1 raw data. Let $w$ and $v$ be the indicators of the information treatment and of the deepfake video, and let $z$ be the vector of covariates. I write the fitted model as

m(w, v, z) = a + b w + c v + d w v + \gamma^{\mathsf T} z .

For fixed $z$ , I have

m(1, v, z) - m(0, v, z) = b + d v, \qquad m(w, 1, z) - m(w, 0, z) = c + d w .

Table OA5 reports $b$ , which is the first contrast at $v = 0$ , and Table OA6 reports $c$ , which is the second contrast at $w = 0$ . Suppose that a fraction $q$ of the target population has $v = 1$ . Then the first contrast, averaged over $v$ , is $(1 - q) b + q (b + d) = b + q d$ . Similarly, the second contrast averaged over $w$ is $c + p d$ , where $p$ is the fraction with $w = 1$ . The main coefficient $b$ equals the averaged effect only if $q d = 0$ , and $c$ only if $p d = 0$ . For example, if the four cell means for $(w, v) = (0,0), (1,0), (0,1), (1,1)$ are $0, 0, 0, 2$ , then $b = 0$ and $d = 2$ , and the average with $q = 1/2$ is $1$ .

The recalculation reproduces all 18 printed entries from the raw data, in the sample of 1,252 participants that remains after the exclusion stated in the notes. The fitted interaction coefficients are $-0.153$ for favorability, $-0.065$ for vote and $-0.041$ for essential oils.

The main coefficients printed in Tables OA5 and OA6, beside the averages $b + d/2$ and $c + d/2$ from the same models. Each entry is an estimate with its interval, and the intervals for the averages apply the paper’s procedure for the bounds to those contrasts.
Contrast	Outcome	Main coefficient	Average, equal weights
Information	Favor	$-0.219$ $[-0.732, 0.248]$	$-0.295$ $[-0.670, 0.063]$
Information	Vote	$-0.102$ $[-0.311, 0.079]$	$-0.134$ $[-0.285, 0.006]$
Information	Oils	$-0.251$ $[-0.752, 0.263]$	$-0.271$ $[-0.652, 0.109]$
Deepfake	Favor	$0.188$ $[-0.322, 0.669]$	$0.111$ $[-0.254, 0.479]$
Deepfake	Vote	$0.049$ $[-0.162, 0.235]$	$0.017$ $[-0.133, 0.159]$
Deepfake	Oils	$-0.137$ $[-0.634, 0.388]$	$-0.157$ $[-0.531, 0.230]$

Every interval in the table contains zero. In the complete cases, the averaged effects of the information treatment have $p$ -values of $0.004$ , $0.004$ and $0.014$ . These tests leave out the missing outcomes that the bounds are meant to cover, and they do not overturn the reading of “no impact” given above.

What survives

All 18 entries stand as main coefficients with their bounds, and so does the reading that no difference is statistically significant with the bounds. The main results on disbelief in the video, the Study 2 results and the pooled regressions for favorability and vote are separate calculations and do not use these entries. What changes is the labelling: each table reports the effect of one treatment at the reference level of the other, and neither reports the averaged contrasts of the pre-analysis plan.

Corrected

Table OA5. Title: Study 1 – Information about deepfakes has no statistically significant effect, for the real video, on favorability, vote choice, or support for essential oils in medicine (but we are underpowered). Row label: Information Treatment Effect for the real video (deepfake $= 0$ ).

Table OA6. Title: Study 1 – Deepfakes have no statistically significant effect, without information about deepfakes, on favorability, vote choice, or support for essential oils in medicine (but we are underpowered). Row label: Deepfake Treatment Effect without information about deepfakes (information $= 0$ ).

Note to both tables. Each estimate is a main coefficient of a covariate-adjusted model that contains the information treatment, the video type and their interaction, and the intervals use the bounds of Section 2. Table OA5 compares information with no information for the real video. Table OA6 compares the deepfake with the real video without information. Averaging over the other treatment would define different contrasts; the displayed coefficients need only be labelled explicitly.

Crop of Table OA5 with its title: Study 1, Information about deepfakes has no impact on favorability, vote choice, or support for essential oils in medicine (but we are underpowered). The table has columns Favor, Vote and More Essential Oils in Medicine, and three rows. A red outline surrounds the label of the first row, Information Treatment Effect, whose entries are minus 0.22, minus 0.10 and minus 0.25. The row 95 percent Upper Bound has 0.25, 0.08 and 0.26, and the row 95 percent Lower Bound has minus 0.73, minus 0.31 and minus 0.75. — Appendix, Table OA5, p. 13. The outline marks the row label “Information Treatment Effect”, which does not say that the effect is taken for the real video.

Crop of Table OA6 with its title: Study 1, Deepfakes have no impact on favorability, vote choice, or support for essential oils in medicine (but we are underpowered). The table has columns Favor, Vote and More Essential Oils in Medicine, and three rows. A red outline surrounds the label of the first row, Deepfake Treatment Effect, whose entries are 0.19, 0.05 and minus 0.14. The row 95 percent Upper Bound has 0.67, 0.24 and 0.39, and the row 95 percent Lower Bound has minus 0.32, minus 0.16 and minus 0.63. — Appendix, Table OA6, p. 14. The outline marks the row label “Deepfake Treatment Effect”, which does not say that the effect is taken without the information treatment.

P034-N001 Minor

Wittenberg et al. (2021) is not an experiment on deepfakes and does not support the statement that such experiments found no evidence that people believe manipulated videos.

Introduction, first paragraph of p. 2 (PDF p. 2), sentence beginning “However, recent randomized experiments”; reference entry on p. 14 (PDF p. 14)

As printed

p. 2: “However, recent randomized experiments on the impacts of deepfakes in American politics have found no evidence that people believe the content of the manipulated videos (Wittenberg et al. 2021; Vaccari and Chadwick 2020). Barari, Lucas, and Munger (2021) found that deepfakes were persuasive, but that their effects were comparable to those of textual misinformation.”

p. 14, reference list: Wittenberg, Chloe, Ben M Tappin, Adam J Berinsky, and David G Rand. 2021. “The (minimal) persuasive advantage of political video over text.” Proceedings of the National Academy of Sciences 118 (47).

The problem

The sentence attributes to two studies the finding that randomized experiments on deepfakes found no evidence that people believe the manipulated videos. The first of the two, Wittenberg et al. (2021), did not study deepfakes. Its experiments compared ordinary videos, on politics and on COVID-19, with transcripts and with controls. Its Discussion (p. 5) describes the subject as “belief in unmanipulated content” and says that “future work should explore whether our results replicate in the context of misinformation, including deepfakes”. One of the two citations therefore does not support the sentence.

The second citation is an experiment on a deepfake, and its result is narrower than the sentence. Vaccari and Chadwick (2020) showed British respondents a deepfake of American political figures, with or without an educational reveal. They report that 16% of the respondents were deceived, that the difference in deception between the versions was not statistically significant, and that uncertainty was higher without the reveal. The design has no control group (p. 4: “Our study does not include a control group”).

Evidence a check of the printed page

The evidence is a check of the cited sources. I read Wittenberg et al. (2021) in the publisher’s typeset version posted by one of its authors (PNAS 118(47), seven pages), and Vaccari and Chadwick (2020) in the version of record (Social Media + Society 6(1), pp. 1–13). The passages quoted above are on p. 5 of the first and on p. 4 of the second. On p. 6 of the second, the analysis reads “A smaller, though by no means negligible, group (16%) was deceived”.

What survives

The two experiments of the paper, their estimates and their bounds do not use the cited findings. What changes is the summary of earlier work in the motivation, which has to be narrowed. I did not establish that the phrase “prior research has found that such videos have minimal effects” in the abstract is wrong, or that the comparison with Vaccari and Chadwick on p. 7 is wrong. I could not obtain the 2021 preprint of Barari, Lucas and Munger, and the sentence about it was not checked against its text.

Corrected

p. 2: “However, a recent randomized experiment on a deepfake of American political figures found that viewers were not significantly more likely to be deceived when the deepfake was shown without an educational reveal than with one, although they were more uncertain (Vaccari and Chadwick 2020). Barari, Lucas, and Munger (2021) found that deepfakes were persuasive, but that their effects were comparable to those of textual misinformation.”

If Wittenberg et al. (2021) is cited, it should be cited separately, as a comparison of ordinary political video with text and not as an experiment on deepfakes. The comparison on p. 7 can stay qualitative; it is clearer if it names the comparator of that study, the same deepfake shown with the reveal.

Crop of six lines of the Introduction. Two red boxes mark the citation '(Wittenberg et al. 2021', which is split over two lines, in the sentence 'recent randomized experiments on the impacts of deepfakes in American politics have found no evidence that people believe the content of the manipulated videos (Wittenberg et al. 2021; Vaccari and Chadwick 2020).' — Introduction, first paragraph of p. 2 (PDF p. 2). The marked citation is given, together with Vaccari and Chadwick (2020), for experiments on deepfakes that found no evidence of belief in the manipulated videos.

Crop of one entry of the reference list, which begins 'Wittenberg, Chloe, Ben M Tappin, Adam J Berinsky, and David G Rand. 2021.' A red box marks the title '(minimal) persuasive advantage of political video over text', which is followed by 'Proceedings of the National Academy of Sciences 118 (47)' and a DOI link. — Reference list, p. 14 (PDF p. 14). The marked title of the cited article describes a comparison of political video with text.

P034-N002 Minor

Pennycook, Bear, Collins and Rand (2020) does not report that warnings lower belief in true headlines; of the two works cited for that statement, only Clayton et al. (2020) supports it.

Introduction, third paragraph of p. 2 (PDF p. 2); Section 4, first sentence, p. 9 (PDF p. 9); reference entry on p. 13 (PDF p. 13)

As printed

p. 2: “Official warnings about fake news similarly induce participants to disbelieve true headlines (Clayton et al. 2020; Pennycook, Bear, et al. 2020).”

p. 9: “Our findings add to the growing body of literature on textual misinformation, which has consistently found that warnings about fake news may help readers reject misinformation but may have the undesirable effect of increasing Americans' disbelief in true news stories (e.g., Pennycook, Bear, et al. 2020; Clayton et al. 2020).”

p. 13, reference list: Pennycook, Gordon, Adam Bear, Evan T Collins, and David G Rand. 2020. “The implied truth effect: Attaching warnings to a subset of fake news headlines increases perceived accuracy of headlines without warnings.” Management Science 66 (11): 4944–57.

The problem

Both sentences cite Pennycook, Bear, Collins and Rand (2020) for the finding that warnings about fake news lower belief in true headlines. On belief, that article reports the opposite. In its Study 1, the proportion rating true headlines accurate was .542 when warnings were attached to a subset of the false headlines and .530 in the control (p. 4948): “headlines that were not tagged in the warning treatment were rated as more accurate than those in the control”. The article does report one adverse result for true headlines, in its Study 2 (p. 4953): untagged true headlines were shared at a rate of 35.2% under warnings plus verification and 40.1% in the control. That outcome is sharing and not belief. The other citation supports the statement: the abstract of Clayton et al. (2020) says that “exposure to a general warning decreased belief in the accuracy of true headlines”. The phrase “has consistently found” on p. 9 thus rests on one of the two works that it cites.

Evidence a check of the printed page

The evidence is a check of the cited sources. I read the version of record of Pennycook et al. (Management Science 66(11), pp. 4944–4957), including the methods and results of both of its studies. For Clayton et al. (Political Behavior 42, pp. 1073–1095) I read the publisher’s abstract only, and I did not obtain the full text.

What survives

The analogy with textual misinformation stands on Clayton et al. (2020), and no design, estimate or test in the paper depends on the citation. The check does not show that the wider literature disagrees on this point; I did not review it.

Corrected

p. 2: “Official warnings about fake news similarly induce participants to disbelieve true headlines (Clayton et al. 2020).”

p. 9: “Our findings add to the growing body of literature on textual misinformation, which has found that general warnings about fake news can reduce belief in both false and true headlines (e.g., Clayton et al. 2020).”

If Pennycook, Bear, et al. (2020) is kept, it should be described separately: when warnings were attached to a subset of false headlines, the headlines without warnings were rated as more accurate.

Crop of five lines of the Introduction. A red box marks the citation 'Pennycook, Bear, et al. 2020' in the sentence 'Official warnings about fake news similarly induce participants to disbelieve true headlines (Clayton et al. 2020; Pennycook, Bear, et al. 2020).' — Introduction, third paragraph of p. 2 (PDF p. 2). The marked citation is one of the two given for the statement that warnings lead participants to disbelieve true headlines.

Crop of the first six lines of Section 4. Red boxes mark the words 'consistently found' and the citation 'Pennycook, Bear, et al. 2020' in the sentence 'Our findings add to the growing body of literature on textual misinformation, which has consistently found that warnings about fake news may help readers reject misinformation but may have the undesirable effect of increasing Americans' disbelief in true news stories (e.g., Pennycook, Bear, et al. 2020; Clayton et al. 2020).' — Section 4, first sentence, p. 9 (PDF p. 9). The marked words “consistently found” and the marked citation repeat the attribution.

Crop of one entry of the reference list, which begins 'Pennycook, Gordon, Adam Bear, Evan T Collins, and David G Rand. 2020.' Two red boxes mark the words 'increases perceived accuracy of headlines without warnings', split over two lines, at the end of the title 'The implied truth effect: Attaching warnings to a subset of fake news headlines increases perceived accuracy of headlines without warnings.' — Reference list, p. 13 (PDF p. 13). The marked words of the title of the cited article state an increase in the perceived accuracy of headlines without warnings.

P034-N010 Minor

In the Total column of Table OA3, the percentage for “Subjective Evaluation of Character” should be 9.2%, not 9.3%, and the percentage for “Stance on Other Policies” should be 0.8%, not 0.9%.

Online Appendix, Table OA3, column “Total”, rows “Subjective Evaluation of Character” and “Stance on Other Policies”, p. 11 (Appendix PDF p. 11)

See also under Formal statements (Lean): P034-T5

As printed

Table OA3: Study 2 - Hand-coded Romney facts across conditions. The two rows at issue and the last row, each with its row of counts.
	No Info about Deepfakes	Info about Deepfakes	Total
Subjective Evaluation of Character (e.g., trusworthy, traitor, etc.)	9.2%	9.3%	9.3%
N	266	268	534
Stance on Other Policies	0.8%	0.9%	0.9%
N	24	25	49
Total	100%	100%	100%
N	2,898	2,877	5,775

The problem

Each percentage in the table is the count in its row divided by the count at the foot of its column, rounded to one decimal place. In the Total column the counts are 534 and 49. Dividing them by 5,775, I find 9.2% and 0.8%. The table prints 9.3% and 0.9%.

Evidence a recalculation from the source

The evidence is a recalculation from the printed counts in exact decimal arithmetic. I have

\frac{534}{5775}\times 100 = 9.246753\ldots, \qquad \frac{49}{5775}\times 100 = 0.848484\ldots,

and at one decimal place the first value rounds to $9.2$ and the second to $0.8$ . The other 37 percentages of the table agree with their counts, and the counts of each column sum to the printed totals 2,898, 2,877 and 5,775.

What survives

Only these two cells change. The counts, the other percentages of Table OA3 and every analysis of the paper are unaffected.

Corrected

Table OA3, the two rows as corrected.
	No Info about Deepfakes	Info about Deepfakes	Total
Subjective Evaluation of Character (e.g., trusworthy, traitor, etc.)	9.2%	9.3%	9.2%
N	266	268	534
Stance on Other Policies	0.8%	0.9%	0.8%
N	24	25	49

Each percentage in the Total column is the count in its row divided by 5,775, the count at the foot of the column, rounded to one decimal place.

Crop of the bottom of Table OA3. The rows shown, with percentages in the three columns and counts below them, are Stance on Other Policies (0.8, 0.9, 0.9 percent; N 24, 25, 49), Is/Was Pro-Life (0.3, 1.4, 0.8 percent; N 9, 39, 48), Believe Video is Real (0.3, 0.5, 0.4 percent; N 9, 15, 24), Believe Video is Fake (0.2, 0.1, 0.2 percent; N 5, 4, 9) and Total (100, 100, 100 percent; N 2,898, 2,877, 5,775). A red outline surrounds the percentage 0.9 percent in the Total column of the first row. — Online Appendix, Table OA3, p. 11: the last four categories and the foot of the table. The outline marks the Total percentage of “Stance on Other Policies”, printed as 0.9% above the count 49; the last row gives the column totals 2,898, 2,877 and 5,775.

P034-N011 Minor

The Appendix says that Study 2 had no deviations from its pre-analysis plan, but the analyses before and after the election drop nine eligible respondents whom the amended plan keeps (1,830 analyzed instead of 1,839), and the cleaning code drops rows with duplicated IP addresses without requiring that they reached randomization, a rule that the plan does not state.

Appendix, Section 2, “Study 2”, p. 5 (PDF p. 5 of the Appendix); Tables OA11 and OA12, Appendix pp. 16–17; Section 3.1, p. 9 (PDF p. 9); amended pre-analysis plan for Study 2, Section 9 (PDF p. 6 of the plan, whose pages are not numbered); replication files, lines 334–336 of 3-analysis_FINAL.do and lines 23–30 and 195–196 of 2-study2_data_cleanup_FINAL.do

As printed

Appendix p. 5, Section 2, under “Study 2”: “There were no deviations from the PAP in Study 2.”

Amended pre-analysis plan for Study 2, Section 9: “We exclude all participants who satisfy any one of the following conditions:” The second of the four conditions is “Have duplicate IP addresses and reached randomization.” The section continues: “For SQ3_election1 and SQ4_election2, we exclude respondents who:” with the two conditions “Take any portion of the survey between 8am and midnight November 3 $^{\text{rd}}$ ” and “Or take any portion of the survey after 8am November 10 $^{\text{th}}$ ”.

3-analysis_FINAL.do, lines 334–336:

reg deepfake i.tr_warn i.post_election_ind $covars if romney_party==1 ///

& study==2 & (stata_start==mdy(11,1,2020) | stata_start==mdy(11,2,2020) ///

| stata_start==mdy(11,8,2020) | stata_start==mdy(11,9,2020)), robust

2-study2_data_cleanup_FINAL.do, lines 23–25:

duplicates tag ip, gen(dup)

tab dup

drop if dup>0

The problem

The code of Study 2 departs from the amended pre-analysis plan in two places, and the Appendix reports no deviation.

The first departure concerns the analyses before and after the election, which are Tables OA11 and OA12 and the sentence of Section 3.1 on p. 9. The plan excludes a respondent who took any portion of the survey between 8am and midnight on November 3, or after 8am on November 10. The code keeps a respondent only if the survey was started on November 1, 2, 8 or 9; the regressions behind the two tables, on lines 728–766 of 4-Supplementary-Materials_FINAL.do, carry the same restriction. Of the 1,925 respondents in the analysis sample, the code drops 95 and the plan drops 86. The nine respondents whom only the code drops took the survey outside the two excluded periods. Six started and finished on November 3 before 8am, one took it on the evening of November 4, and two started and finished on November 10 before 8am. The code keeps no respondent whom the plan excludes. The tables are computed on 1,830 respondents, and the plan specifies 1,839.

The second departure concerns duplicate IP addresses. The plan excludes participants who “Have duplicate IP addresses and reached randomization.” After removing previews and rows without respondent identifiers, the code removes every remaining row whose IP address occurs more than once, whether or not it reached randomization. At that step 176 rows have a duplicated address; 62 of them reached randomization and 114 did not. If duplicates are counted among these rows under both rules, then the two rules leave the same 1,925 respondents in the analysis sample. They do not leave the same cleaned file. Nine of the 114 rows pass every other step of the cleaning. The code replaces each missing value of household income by the mean over the cleaned file, and it does so before it selects the analysis sample. That mean is $387/37\approx 10.459$ without the nine rows and $29877/2858\approx 10.454$ with them, and 75 of the 1,925 respondents receive it. The plan does not say whether a duplicate is counted among all rows or only among those that reached randomization. On the second reading, 13 more respondents would enter the analysis sample.

Evidence a recalculation from the source

The evidence is a recalculation from the public Study 2 data. The relevant cleaning and regression calculations were translated from Stata into Python, using the supplied cleaned data for other covariates; Stata was not run. The translated cleaning gives the 2,991 respondents of the cleaned data and the 1,925 of the analysis sample, 966 without and 959 with information. With the rules of the code, I reproduce all 24 printed entries of Tables OA7, OA11 and OA12. (The tables stored in the replication files differ from the printed ones; see P034-E019.) All 1,925 surveys have recorded start and end times; no endpoint lies on a boundary, and no survey spans the whole November 3 exclusion period. I then computed Tables OA11 and OA12 again with the exclusions of the plan.

Tables OA11 and OA12 with the sample of the code and with the sample of the amended plan. Each entry is the coefficient with its interval of $\pm 1.96$ robust standard errors, as in the Appendix.
Table and column	Code, $N=1{,}830$	Plan, $N=1{,}839$
OA11, Deepfake?	$-0.140$ $[-0.272,\,-0.008]$	$-0.134$ $[-0.266,\,-0.002]$
OA11, Unconvinced	$-0.047$ $[-0.116,\,0.022]$	$-0.047$ $[-0.116,\,0.022]$
OA11, Don’t Know if Real	$-0.009$ $[-0.053,\,0.036]$	$-0.005$ $[-0.050,\,0.039]$
OA12, Deepfake?	$0.123$ $[-0.140,\,0.386]$	$0.133$ $[-0.129,\,0.395]$
OA12, Unconvinced	$-0.059$ $[-0.195,\,0.077]$	$-0.057$ $[-0.193,\,0.079]$
OA12, Don’t Know if Real	$0.029$ $[-0.059,\,0.116]$	$0.027$ $[-0.061,\,0.114]$

For the first row, the $p$ -value is $0.038$ with the sample of the code and $0.047$ with the sample of the plan. At the printed precision, six entries change with the sample of the plan. In Table OA11 the first column becomes $-0.13$ with upper bound $0.00$ (printed $-0.14$ and $-0.01$ ). In Table OA12 the first column becomes $0.13$ with lower bound $-0.13$ (printed $0.12$ and $-0.14$ ), the lower bound of the second column becomes $-0.19$ (printed $-0.20$ ), and the upper bound of the third becomes $0.11$ (printed $0.12$ ).

For the second departure, I repeated the cleaning and removed only the duplicated rows that reached randomization. The effect of the information on the main outcome of Study 2 is $0.263595$ with either mean of income, and no printed entry of Tables OA7, OA11 and OA12 changes.

What survives

The sentence of Section 3.1 stands: the difference between respondents surveyed before and after the election has $p<0.05$ with either sample. Every interval of Table OA12 contains zero with either sample. The printed entries of Tables OA11 and OA12 are correct for the respondents who started the survey on November 1, 2, 8 or 9. The main effect of Study 2 remains $0.26$ with $p<0.001$ under either mean of income; I did not recompute every other coefficient or results under the alternative duplicate-counting rule. Study 1 does not pass through this code, and the other analyses of Study 2 and the pooled analyses do not use the restriction by date.

Corrected

Either the Appendix reports the two departures, or the code follows the plan.

To keep the printed tables, Appendix p. 5 can read: “Study 2 had two deviations from the PAP. First, the analyses for SQ3_election1 and SQ4_election2 (Tables OA11 and OA12) include only respondents who started the survey on November 1, 2, 8 or 9, 2020. This rule excludes nine respondents whom the exclusions of the PAP retain, so that these analyses use 1,830 respondents and not 1,839. Second, after removing previews and responses without respondent identifiers, we removed every remaining response whose IP address occurred more than once, whether or not it reached randomization, before imputing missing household income with the mean of the cleaned sample.”

To follow the plan for these data, let $s_i$ and $e_i$ be the recorded start and end of the survey of respondent $i$ , in 2020. The restriction to four start dates is replaced by the exclusion of respondent $i$ when

\colorbox{#ffe58a}{$\displaystyle \bigl(s_i<\text{November 4, 0:00 and } e_i\ge\text{November 3, 8:00}\bigr)\quad\text{or}\quad e_i>\text{November 10, 8:00}$}.

At the precision shown, Tables OA11 and OA12 then take the values in the last column of the table above. In the cleaning, keep the existing population used to count duplicates, and identify randomized rows from the recorded treatment branches. Restrict drop if dup>0 to those randomized rows, and impute household income after that step. Restrict the following isid rid and isid ip uniqueness checks to randomized rows as well. The Appendix should also say among which rows a duplicate is counted.

Crop of the end of the passage on Study 1 and the passage on Study 2 in Section 2 of the Appendix. The last paragraph on Study 1 reads: We also had two minor deviations from the PAP. To be consistent with Study 2's PAP, we 1) dropped the one IP address that was randomized three times (there were no other IP duplicates that watched our video) and 2) used robust standard errors. These modifications have negligible impacts on the reported results. Under the heading Study 2, a red box marks the sentence 'There were no deviations from the PAP in Study 2.' — Appendix, Section 2, p. 5 (PDF p. 5 of the Appendix). The marked sentence says that Study 2 had no deviations from its pre-analysis plan.

Crop of Section 9, Exclusions, of the amended plan. After the sentence 'We exclude all participants who satisfy any one of the following conditions:' come four bullets: did not consent or meet the eligibility criteria; 'Have duplicate IP addresses and reached randomization.', marked with a red box; failed any one of the audio, visual, and attention checks; self-reported age was not within one year of their age according to Lucid. After a sentence on participants who could not identify Mitt Romney's political party, the text reads 'For SQ3_election1 and SQ4_election2, we exclude respondents who:' followed by two bullets, each marked with a red box: 'Take any portion of the survey between 8am and midnight November 3rd' and 'Or take any portion of the survey after 8am November 10th'. — Amended pre-analysis plan for Study 2, Section 9, “Exclusions” (PDF p. 6 of the plan). The marked lines are the exclusion for duplicate IP addresses, which names respondents who reached randomization, and the two exclusions by time for the analyses before and after the election.

Crop of Table OA11 with its title: Table OA11: Study 2 - Some evidence of more distrust immediately before a major election. The columns are Deepfake?, Unconvinced and Don't Know if Real. The row Post Election Indicator reads minus 0.14, minus 0.05, minus 0.01. The row 95 percent Upper Bound reads minus 0.01, 0.02, 0.04. The row 95 percent Lower Bound reads minus 0.27, minus 0.12, minus 0.05. Red boxes mark the first two entries of the column Deepfake?, minus 0.14 and minus 0.01. — Appendix, Table OA11, p. 16 (PDF p. 16 of the Appendix). The table is computed on the 1,830 respondents who started the survey on November 1, 2, 8 or 9. With the exclusions of the amended plan, which leave 1,839 respondents, the two marked entries become $-0.13$ and $0.00$ .

Clarity notes

Imprecise or ambiguous wording whose intended meaning is correct; not an error.

P034-E002 Clarity note

“Unable to discriminate” and “did not enhance” mean that no difference and no improvement were detected; the tests reported do not establish that there was none.

Abstract, p. 1 (PDF p. 1); Introduction, p. 3 (PDF p. 3); footnote 15, p. 7 (PDF p. 7); Appendix, Section 3 and the title of Figure OA1, p. 6 (PDF p. 6 of the Appendix)

As printed

Abstract, p. 1: “We conducted two online survey experiments, and found that informing participants about deepfakes did not enhance participants' ability to successfully spot manipulated videos but consistently induced them to believe the videos they watched were fake, even when they were real.”

p. 3: “We found that participants were unable to discriminate between real and deepfaked videos even when they were informed about the existence of deepfakes.”

p. 7, footnote 15: “Participants were not able to discern between a real video and a deepfake version of the same video without the information treatment (see Section 3 of the Appendix for more details).”

Appendix p. 6: “One key question is whether American voters can discern between deepfakes and real videos without being informed about deepfakes. In Figure OA1, we show that participants were unable to do so.”

Appendix p. 6, title: “Figure OA1: Voters were unable to discriminate between a real video and a deepfake (Study 1)”

Intended meaning

The passages summarize tests of differences in mean ratings for the two videos of Study 1. They mean that the study detected no significant difference between the ratings of the real video and of its deepfake version, with or without the warning, and no significant improvement due to the warning. This is correct, and p. 8 states it: the interaction of the two treatments “is not statistically significant”, and its estimate has the opposite sign from an improvement. Recalculating from the Study 1 data, I obtain the following values. Without the warning, the adjusted difference (deepfake minus real) in the belief that the video is a deepfake is $0.062$ , with $p=0.59$ . Its complete-case 95% interval is $[-0.164,0.287]$ , and its interval with Manski bounds is $[-0.384,0.469]$ , as in Figure OA1. With the warning, the difference is $-0.130$ , with $p=0.26$ and interval $[-0.478,0.215]$ with Manski bounds. The interaction is $-0.214$ , with $p=0.19$ as in footnote 17 and complete-case interval $[-0.534,0.107]$ .

Read literally, “we show that participants were unable” says more, namely that the tests establish that participants could not tell the videos apart. A test that does not reject does not establish this, and the intervals above contain positive values. For example, suppose that every rating is $\pm1$ , and that the probability of $+1$ is $2/5$ for the real video and $3/5$ for the deepfake without the warning, and $3/10$ and $7/10$ with it. Then the mean difference is $2/5$ without the warning and $4/5$ with it. With ten participants in each of the four cells, the event that every cell has five ratings of each sign has positive probability (about $0.0004$ ). On this event every sample difference and the sample interaction are zero, with $p=1$ . This example does not show that the study’s participants could discriminate or that the warning helped them. No estimate, test or conclusion of the paper changes.

Corrected

Abstract: “We conducted two online survey experiments. For the selected Study 1 videos, we did not detect an improvement in discrimination between the real and manipulated videos when participants were informed about deepfakes; the information consistently induced them to believe the videos they watched were fake, even when they were real.”

p. 3: “For the selected Study 1 videos, we did not detect a statistically significant difference in participants' ratings of the real and deepfaked videos, with or without information about deepfakes.”

p. 7, footnote 15: “Without the information treatment, we did not detect a statistically significant difference in participants' ratings of the real video and its deepfake version (see Section 3 of the Appendix for more details).”

Appendix p. 6: “In Figure OA1, we report no statistically significant difference between the real and deepfake conditions for the selected stimulus.”

Appendix p. 6, title: “Figure OA1: No statistically significant real/deepfake difference in participants' ratings (Study 1, without deepfake information)”

These are results of tests of a difference and not of an equivalence analysis. The statements of the paper about power and about generalization stay as they are.

Crop of five lines of the Introduction. A red box marks the words 'were unable to discriminate' in the sentence 'We found that participants were unable to discriminate between real and deepfaked videos even when they were informed about the existence of deepfakes.' — Introduction, p. 3 (PDF p. 3). The marked words state as an inability what the tests report as a difference that was not detected.

Crop of six lines of the abstract. Three red boxes mark the clause 'did not enhance participants' ability to successfully spot manipulated videos', which runs over two lines, in the sentence 'We conducted two online survey experiments, and found that informing participants about deepfakes did not enhance participants' ability to successfully spot manipulated videos but consistently induced them to believe the videos they watched were fake, even when they were real.' — Abstract, p. 1 (PDF p. 1). The marked clause summarizes the test of the interaction on p. 8, which was not significant.

Crop of the last four footnote lines of the page. A red box marks the words 'were not able to discern' in footnote 15: 'Participants were not able to discern between a real video and a deepfake version of the same video without the information treatment (see Section 3 of the Appendix for more details).' — Footnote 15, p. 7 (PDF p. 7). The marked words make the same statement for participants who saw no warning.

Crop of the first four lines of Section 3 of the Appendix. Two red boxes mark the words 'we show that participants were unable to do so.', split over two lines, after the sentence 'One key question is whether American voters can discern between deepfakes and real videos without being informed about deepfakes.' — Appendix, Section 3, p. 6 (PDF p. 6 of the Appendix). The marked sentence says that Figure OA1 shows the inability.

Crop of the title of Figure OA1 and the line above it. A red box marks the words 'were unable to discriminate' in the title 'Figure OA1: Voters were unable to discriminate between a real video and a deepfake (Study 1)'. — Appendix p. 6, title of Figure OA1. The marked words repeat the statement in the title.

Formal statements (Lean)

Formal results: 5 whole; 0 restricted; 0 refuted; 0 omitted.

Coverage concerns the identified claim under its stated assumptions. A checked proof establishes the formal statement; correspondence to the published text was reviewed by AI. Repository access is currently required to open the source links.

Each result links to its current formal statement and proof. “Whole” covers the identified claim under its assumptions; “restricted” covers the stated narrower scope; “refuted” gives a counterexample to the identified assertion; “omitted” has no current theorem. Any named published theorem used as a premise remains an assumption. Browse the formal results for this work.

P034-T1

Amended Study 2 pre-analysis plan, Section 8 (Missing Data): the rule “If there is no differential attrition, we conclude that missingness occurs at random”, repeated as a justification in the notes to Appendix Figures OA2a and OA3. The witness is the validator's construction and is not in the paper.

Amended Study 2 pre-analysis plan (staged as code/pap/DF_Study2_PAP_Amended.pdf), PDF p. 5 of 6 (pages unnumbered), Section 8, second paragraph, ending on PDF p. 6; online Appendix p. 9 (PDF p. 9), note to Figure OA2a, and p. 18 (PDF p. 18), note to Figure OA3; related sentence in the article, p. 6 (PDF p. 6), Section 2.2

amended Study 2 pre-analysis plan, Section 8, PDF pp. 5-6; Appendix p. 9, Figure OA2a note, and p. 18, Figure OA3 note. The specified randomized finite population has equal completion rates but outcome-dependent missingness and a biased complete-case OLS coefficient, including the nonrejecting sample and the observationally equivalent independent-completion population with a different treatment effect. Probabilities and moments use exact finite rational sums; both OLS uniqueness assertions quantify over real coefficient vectors.

Historical component credits

Counterexample proved AronowAudit.P034.missingness_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'adjudication', 'decision': 'rejected', 'what': 'P034_Attrition.lean:353 bundles the specified witness in T1(i)-(ix): normalized positive laws, randomization, equal completion rates, outcome-dependent selection, the nonzero complete-case contrast and adjusted coefficient, the nonrejecting sample, and observational equivalence. Boolean coding and rational finite sums exactly represent this fixed binary construction, and the cell multiplicities at P034_Attrition.lean:210 preserve all 400 sample records relevant to the claimed statistics. Although the explicit coefficient quantifiers at P034_Attrition.lean:355 and :359 use rationals, the included determinant identity at P034_Attrition.lean:174 and sample scaling at :263 ensure the same unique solution over the reals, so no additional coefficient solutions are left open. The target explicitly requests this validator-supplied construction, and no material narrowing or omitted conclusion remains.'}

Formal result: whole

Formal statement · Proof · Definitions

Assumptions and verification

Claim revision: catalog:923ad8c1abf4a0a2f5e774ff85cd2d502c625295d778c033aeab2259e2a40e1c

Standing assumptions: The article is an empirical report of two survey experiments. It contains no theorem, no displayed formula and no formal model; it states no potential-outcome notation and no identification assumption. The setting, as printed (read from the page images; in the article and in the online Appendix the printed page number equals the PDF page number): Study 1 design (article p. 4): “We randomized participants to receive either the real video or the deepfake version and either information about deepfakes or no additional information in a 2x2 fully factorialized experiment. The four conditions are summarized in Table 1 on the next page.” Table 1 (article p. 5), “Description of Study 1 treatment arms”: rows Real Video (N=696) and Deepfake Video (N=700); columns No Information (N=696) and Information about Deepfakes (N=700); cells (1) N=347, (2) N=349, (3) N=349, (4) N=351. Study 1 attrition and bounds (article p. 5): “We found that survey attrition varied significantly across treatment conditions (Pearson $\chi^2$(3) = 11.3, p = 0.01): 4.9% of participants in the No Information + Real Video condition did not finish the survey, compared to 1.8% in the three remaining conditions combined. Because our pre-analysis plan did not explicitly stipulate how we would address differential attrition, we used Manski-type worst-case bounds (Manski 2003), which require only that the support of the outcome is bounded when constructing our confidence intervals. These bounds are considered the gold-standard approach when analyzing experiments with attrition (Gerber and Green 2012). Additional details are provided in Section 2 of the Appendix.” Footnote 11 (article p. 5): “The “best” and “worst” cases used to estimate the confidence interval are also covariate adjusted and use robust standard errors.” Study 1 bounds, method (Appendix p. 5, Section 2, Study 1): “The major deviation is the use of Manski bounds for confidence intervals and statistical significance. For convenience, we provide a brief summary of the method here (for more details, please refer to Manski (2003)). We impose the distribution of outcomes that least favors our hypothesis onto all missing outcomes (i.e., the survey instrument maximum for all missing outcomes in the control and the survey instrument minimum for all missing outcomes in treatment), run a covariate-adjusted OLS regression with robust standard errors over all outcomes, and use the lower bound of the resulting 95% confidence interval as the lower bound of our reported 95% CI. We do the same with a best-case distribution of missing outcomes to estimate the upper bound of our reported 95% CI.” Study 2 design (article p. 6): “The design of this study was nearly identical to that of Study 1 with the following two modifications. First, Study 2 had only two conditions, Information about Deepfakes and No Information (see Table 2 on the next page), as we did not create a deepfake of Mitt Romney given ethical considerations. Second, our analysis was restricted to participants who knew Romney was a Republican pre-treatment, as Romney’s (formerly) pro-choice position would not appear surprising to participants with no knowledge of his party affiliation. In Study 2, we found no evidence of differential attrition: survey completion rates did not differ significantly across treatment arms.” Table 2 (article p. 7), “Description of Study 2 treatment arms”: Real Video (N=1,925); No Information (N=966); Information about Deepfakes (N=959). Appendix p. 5, Section 2, Study 2: “There were no deviations from the PAP in Study 2.” Primary outcome (article pp. 6-7): “The primary outcome measure used asked participants the extent to which they agreed with the statement “This video was doctored, manipulated and/or faked by a computer (i.e. it is a ‘Deep Fake’)” on a 7-point agree/disagree scale ranging from strongly disagree (-3) to strongly agree (3).” Study 2 analysis as planned (amended Study 2 pre-analysis plan, staged as code/pap/DF_Study2_PAP_Amended.pdf, PDF p. 5 of 6, pages unnumbered, Section 6 “Analysis”): “The main analysis will use a covariate adjusted OLS regression. The covariates are: Lucid demographic variables, policy stances, ideology, Big Five personality traits, political knowledge, and political interest.” Scope of the targets. None of the headline claims P034-H1 to P034-H7 is a mathematical statement (each is an empirical estimate or an interpretive conclusion), so there is no headline target. No finding is rated above Minor (15 confirmed errors, all Minor), so there is no result_touched_by_finding target. The targets below exist because three confirmed Minor errors have mathematical content: P034-E017 (an invalid inference from equal completion rates to random missingness, with an exact counterexample), P034-A001 (coefficients of an interaction model printed under labels that read as overall effects; the contrast algebra) and P034-N010 (two misrounded percentages). The other confirmed errors are not targets because their corrections change no mathematical statement: P034-E003 (which of two computed p-values is printed for the Study 1 deepfake arm; the corrected value, p about 0.003 under the unfavourable completion, is a recalculation from the raw data, and the printed p<0.001 is correct for the complete-case regression); P034-E005, P034-E006, P034-E007, P034-E009, P034-E011 and P034-E019 (defects of code, hand coding and cached output); P034-E012, P034-E015 and P034-E018 (misprints in labels, cross-references, spelling and references, among them “p<.0369” for p = .0369 in the note to Table OA2); P034-N001 and P034-N002 (citations).

Effective claim (source TeX): [Printed statements] [Amended Study 2 pre-analysis plan, Section 8 “Missing Data”, second paragraph, PDF p. 5 of 6, ending on PDF p. 6] “Since there may be individuals who are randomized but do not finish the survey (and thus do not provide outcome responses), we will check for differential attrition using a chi-squared test (see line 13 in the Stata code). If there is no differential attrition, we conclude that missingness occurs at random and exclude all affected observations from analysis. If there is differential attrition (i.e., chi-squared p-value<.05), we will use inverse probability weights to weight the population of all subjects who passed the attention checks. The missingness propensity score will be fit via a logistic regression of missingness on covariates, treatment, and their interactions. Any propensity scores under .05 will be Winsorized to 0.05.” [Appendix p. 9, note to Figure OA2a] “The rightmost treatment effect graph uses covariate-adjusted standard errors from an OLS regression, as there was no differential attrition.” [Appendix p. 18, note to Figure OA3] “The confidence intervals in Study 2 are estimated using covariate-adjusted standard errors from an OLS regression, as there was no differential attrition.” [Counterexample to the implication “no differential attrition, therefore missingness occurs at random”, read as licensing the complete-case covariate-adjusted OLS coefficient as the treatment effect in the randomized population. The construction is the validator's and is not in the paper.] Let $a=37/50$ and $b=13/50$. Let $\Omega=\{0,1\}^5$ with points $\omega=(u,r_0,r_1,x,t)$ and point masses $$P(u,r_0,r_1,x,t)=\tfrac18\;p_0(u)^{r_0}\,(1-p_0(u))^{1-r_0}\;p_1(u)^{r_1}\,(1-p_1(u))^{1-r_1},$$ where $p_0(0)=a$, $p_0(1)=b$, $p_1(0)=b$, $p_1(1)=a$. On $\Omega$ define the potential outcomes $Y(0)=Y(1)=u$, the potential completion indicators $R(0)=r_0$ and $R(1)=r_1$, the pretreatment covariate $X=x$, the treatment $T=t$, the outcome $Y=Y(T)$ and the completion indicator $R=R(T)$; $Y$ is observed if and only if $R=1$. Claims. (i) $P$ is a probability measure on $\Omega$ and all 32 point masses are strictly positive. (ii) Randomization holds: $T$ is independent of $(Y(0),Y(1),R(0),R(1),X)$, and $P(T=1)=1/2$. (iii) The treatment effect is zero for every unit: $Y(1)-Y(0)=0$ on $\Omega$, so $\tau:=E[Y(1)-Y(0)]=0$. (iv) There is no differential attrition, exactly: $P(R=1\mid T=0)=P(R=1\mid T=1)=1/2$. (v) Missingness depends on the outcome within treatment arm and covariate stratum: for each $x\in\{0,1\}$, $P(R=1\mid T=0,X=x,Y=0)=37/50$ and $P(R=1\mid T=0,X=x,Y=1)=13/50$, while $P(R=1\mid T=1,X=x,Y=0)=13/50$ and $P(R=1\mid T=1,X=x,Y=1)=37/50$. In particular $R$ is not independent of $Y$ given $(T,X)$, and $R(0)$ is not independent of $Y(0)$. (vi) Complete-case means: $E[Y\mid T=0,R=1]=13/50$ and $E[Y\mid T=1,R=1]=37/50$, so the complete-case difference in means is $12/25\neq 0=\tau$. (vii) Complete-case covariate-adjusted OLS: with $Z=(1,T,X)'$, $$G:=E[ZZ'\mid R=1]=\begin{pmatrix}1&1/2&1/2\\ 1/2&1/2&1/4\\ 1/2&1/4&1/2\end{pmatrix},\qquad h:=E[ZY\mid R=1]=\begin{pmatrix}1/2\\ 37/100\\ 1/4\end{pmatrix},$$ $\det G=1/16$, and the unique solution of $G\beta=h$ is $\beta=(13/50,\ 12/25,\ 0)'$. The coefficient on $T$ is $12/25$, not $0$. (viii) A sample in which the plan's test does not reject. Consider the data set of $N=400$ units in which, for each $(t,x)\in\{0,1\}^2$, 50 units have $(T,X)=(t,x)$ and $R=0$ (outcome missing) and 50 units have $(T,X)=(t,x)$ and $R=1$; among the latter, $Y=1$ for 13 units and $Y=0$ for 37 units when $t=0$, and $Y=1$ for 37 units and $Y=0$ for 13 units when $t=1$. Every observed cell of this data set has positive probability under $P$. The $T$-by-$R$ table of counts is $\begin{pmatrix}100&100\\ 100&100\end{pmatrix}$, each expected count $E_{ij}=(\text{row}_i\times\text{column}_j)/N$ equals $100$, and Pearson's statistic $\sum_{i,j}(O_{ij}-E_{ij})^2/E_{ij}$ equals $0$, so its $\chi^2_1$ p-value is $1$ and the test does not reject at level $.05$. The least-squares regression of $Y$ on $(1,T,X)$ over the 200 units with $R=1$ has coefficient vector $(13/50,\ 12/25,\ 0)$. (ix) Observational equivalence with a completely-at-random mechanism. Let $\Omega_B=\{0,1\}^6$ with points $(y_0,y_1,r_0,r_1,x,t)$ whose six coordinates are independent under $P_B$ with $P_B(y_0=1)=13/50$, $P_B(y_1=1)=37/50$ and $P_B(r_0=1)=P_B(r_1=1)=P_B(x=1)=P_B(t=1)=1/2$, and define $Y(0)=y_0$, $Y(1)=y_1$, $R(0)=r_0$, $R(1)=r_1$, $X=x$, $T=t$, $Y=Y(T)$, $R=R(T)$. Under $P_B$, $(R(0),R(1))$ is independent of $(Y(0),Y(1),X,T)$ and $E_B[Y(1)-Y(0)]=12/25$. The observed-data laws under $P$ and $P_B$ coincide: for all $t,x\in\{0,1\}$, the probability of $\{T=t,X=x,R=0\}$ is $1/8$ under both; the probability of $\{T=0,X=x,R=1,Y=1\}$ is $13/400$ and that of $\{T=0,X=x,R=1,Y=0\}$ is $37/400$ under both; the probability of $\{T=1,X=x,R=1,Y=1\}$ is $37/400$ and that of $\{T=1,X=x,R=1,Y=0\}$ is $13/400$ under both. Hence the law of the observed data, and in particular any test of completion rates, cannot distinguish the outcome-dependent mechanism with $\tau=0$ from the completely-at-random mechanism with effect $12/25$.

: SAP.P034.T1Statement

Theorem: SAP.P034.T1

Statement hash: b0a9bed638aed243175161c622a7ec81d55e4ff1c37f8b09dcffdac10bac23b6

Statement definition: SAP.P034.T1Statement

Certificate. 20261103T230208Z

Declaration review: faithful; 2026-11-04T12:53:35Z; binding v2

Evidence scope: Certification checks formal proof objects. Statement fidelity is a model judgment. Author review is separate.

Proof dependencies: Independently extracted

Source and evidence metadata · sap-2026-11-08-assured-release

P034-T2

Amended Study 2 pre-analysis plan, Section 8 (Missing Data), second paragraph, as repaired by Sol; the same repair applies to the clause “as there was no differential attrition” in the notes to Appendix Figures OA2a and OA3

Amended Study 2 pre-analysis plan (staged as code/pap/DF_Study2_PAP_Amended.pdf), PDF p. 5 of 6 (pages unnumbered), Section 8; online Appendix p. 9 (PDF p. 9) and p. 18 (PDF p. 18)

corrected amended Study 2 pre-analysis plan, Section 8, PDF pp. 5-6, and Appendix pp. 9 and 18. Randomization, overlap, finite second moments and independent positive potential completion preserve arm means and covariate laws, identify the OLS treatment coefficient, and give a unique solution when nonsingular; the fixed counterexample shows that equal completion rates do not suffice.

Historical component credits

Proved AronowAudit.P034.corrected_complete_case_identification Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Matches T2(a)–(e) on an arbitrary probability space. The second moments and joint independence assumptions are in the target itself; no identifying result is assumed. The unrestricted normal-equation conclusion is conditional on being a solution, exactly as in T2(d); nonsingularity is required only for the unique-existence conclusion. Reporting instructions are outside the mathematical component. No change after reading the record.

Formal result: whole

Formal statement · Proof · Definitions

Assumptions and verification

Claim revision: catalog:b056c5d9e1dd32fd15c0eef7bf9d9e16d73cd5a33762a6793d206d42618075dd

Effective claim (source TeX): [Printed sentence that is replaced: amended Study 2 pre-analysis plan, Section 8, PDF p. 5] “If there is no differential attrition, we conclude that missingness occurs at random and exclude all affected observations from analysis.” [Sol's corrected text (validated.json, P034-E017, prior_correction.corrected_text; the same repair as smallest_repair), verbatim] “If the chi-squared test does not reject, we will report no detected difference in completion rates across treatment arms. This result does not establish random or ignorable outcome missingness. Complete-case covariate-adjusted OLS requires a separate identifying condition; for example, within the target population, completion under each treatment is independent of potential outcomes and pretreatment covariates, with positive completion probability. The test does not verify that condition. If a suitable identifying condition cannot be justified, we will report attrition bounds or sensitivity analyses.” [Mathematical content of the corrected text, with every hypothesis; notation of P034-T1 and of the validator's derivation.md, sections 2 and 8] [Setting] A probability space $(\Omega,\mathcal F,P)$ carrying a treatment indicator $T\in\{0,1\}$; real potential outcomes $Y(0),Y(1)$ with $E[Y(t)^2]<\infty$; potential completion indicators $R(0),R(1)\in\{0,1\}$; and a vector of pretreatment covariates $X\in\mathbb R^k$ with $E\|X\|^2<\infty$. The realized outcome is $Y=Y(T)$ and the realized completion indicator is $R=R(T)$; $Y$ is observed if and only if $R=1$. The target is $\tau=E[Y(1)-Y(0)]$. [Hypotheses] (H1, randomization) $T$ is independent of $(Y(0),Y(1),R(0),R(1),X)$ and $0<P(T=1)<1$. (H2, the separate identifying condition) for each $t\in\{0,1\}$, $R(t)$ is independent of $(Y(0),Y(1),X)$ and $P(R(t)=1)>0$. [Statement] Under (H1) and (H2): (a) $P(T=t,R=1)=P(T=t)\,P(R(t)=1)>0$ for $t=0,1$, so that conditioning on $\{T=t,R=1\}$ is well defined and $p:=P(T=1\mid R=1)\in(0,1)$. (b) $E[Y\mid T=t,R=1]=E[Y(t)]$ for $t=0,1$; hence the complete-case difference in means is the target, $E[Y\mid T=1,R=1]-E[Y\mid T=0,R=1]=\tau$. (c) For $t=0,1$ the conditional law of $X$ given $\{T=t,R=1\}$ equals the law of $X$; in particular $E[(T-p)X\mid R=1]=0$. (d) Complete-case covariate-adjusted OLS: with $Z=(1,T,X')'$, $G=E[ZZ'\mid R=1]$ and $h=E[ZY\mid R=1]$, every solution $\beta=(\beta_0,\beta_T,\beta_X')'$ of the normal equations $G\beta=h$ has $\beta_T=\tau$; if $G$ is nonsingular the solution is unique. [What the correction removes] (e) (H2) does not follow from (H1) together with equal completion rates $P(R=1\mid T=0)=P(R=1\mid T=1)$: in the model of P034-T1, (H1) holds and both completion rates equal $1/2$, yet (H2) fails, $\tau=0$ and $\beta_T=12/25$.

{Ω : Type*} [MeasurableSpace Ω] : SAP.P034.T2Statement Ω

Theorem: SAP.P034.T2

Statement hash: ffb01dd9b931c32dab381b8f912e01e24fe0a4c87d3a1a38ccec6d01ccf0fde4

Statement definition: SAP.P034.T2Statement

Certificate. 20261103T230208Z

Declaration review: faithful; 2026-11-04T12:53:35Z; binding v2

Evidence scope: Certification checks formal proof objects. Statement fidelity is a model judgment. Author review is separate.

Proof dependencies: Independently extracted

Source and evidence metadata · sap-2026-11-08-assured-release

P034-T3

Tables OA5 and OA6, first rows (“Information Treatment Effect”, “Deepfake Treatment Effect”), read as effects averaged over the other randomized factor. The witness is the validator's construction and is not in the paper.

Online Appendix p. 13 (PDF p. 13), Table OA5, and p. 14 (PDF p. 14), Table OA6; the article, p. 9 (PDF p. 9), Section 3.1, refers the reader to these Appendix sections

online Appendix pp. 13-14, Tables OA5-OA6. The four specified factorial cells have a unique saturated interpolation with both main coefficients zero and both equally weighted contrasts one, separating the averaged-effect interpretation from the printed row labels.

Historical component credits

Counterexample proved AronowAudit.P034.factorial_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'adjudication', 'decision': 'rejected', 'what': "P034_Contrasts.lean:108 states the unique real saturated fit (0,0,0,2) for the specified four cell outcomes and both equal-weight contrasts equal to one while the main coefficients are zero, exactly as T3(i)-(iii) requests. The target expressly presents unique interpolation as equivalent to the four-unit least-squares formulation; zero covariate dimension is part of this particular counterexample, not a restriction of a universally quantified claim. The target also explicitly identifies the coefficient/average identity as an interpretation of the printed labels and attributes the construction to the validator. Consequently the theorem's failure to refute the actual table entries or a literally printed identity is not a fidelity defect in this row."}

Formal result: whole

Formal statement · Proof · Definitions

Assumptions and verification

Claim revision: catalog:810791f0e0586dcdd104a9781995f071cb38346534d8386edce986f6707cc846

Effective claim (source TeX): [Printed] [Appendix p. 13] “Table OA5: Study 1 – Information about deepfakes has no impact on favorability, vote choice, or support for essential oils in medicine (but we are underpowered)”. Columns: Favor; Vote; More Essential Oils in Medicine? Rows: Information Treatment Effect: -0.22, -0.10, -0.25; 95% Upper Bound: 0.25, 0.08, 0.26; 95% Lower Bound: -0.73, -0.31, -0.75. [Appendix p. 14] “Table OA6: Study 1 – Deepfakes have no impact on favorability, vote choice, or support for essential oils in medicine (but we are underpowered)”. Same columns. Rows: Deepfake Treatment Effect: 0.19, 0.05, -0.14; 95% Upper Bound: 0.67, 0.24, 0.39; 95% Lower Bound: -0.32, -0.16, -0.63. Note printed under each table: “As mentioned in the PAP, this analysis excludes participants who are likely supporters of alternative medicine. This is defined as ALL participants EXCEPT those who BOTH oppose government mandates for vaccination AND support government subsidies for acupuncture procedures.” [Counterexample to the identity under which the printed labels would be complete: “in a model containing both treatment indicators and their product, the coefficient on one treatment indicator equals the effect of that treatment averaged over the other treatment”. The construction is the validator's and is not in the paper.] Let $w,v\in\{0,1\}$ and take a 2x2 factorial with cell values $\mu(0,0)=0$, $\mu(1,0)=0$, $\mu(0,1)=0$, $\mu(1,1)=2$. (i) The unique $(a,b,c,d)\in\mathbb R^4$ with $\mu(w,v)=a+bw+cv+dwv$ in all four cells is $(a,b,c,d)=(0,0,0,2)$; equivalently, for the data set of four units, one in each cell, with outcomes $\mu(w,v)$, the least-squares fit of the outcome on $(1,w,v,wv)$ is exact with coefficient vector $(0,0,0,2)$. (ii) The coefficient on $w$ is $b=\mu(1,0)-\mu(0,0)=0$, whereas the effect of $w$ averaged over $v$ with equal weights is $\tfrac12[\mu(1,0)-\mu(0,0)]+\tfrac12[\mu(1,1)-\mu(0,1)]=1$. (iii) The coefficient on $v$ is $c=\mu(0,1)-\mu(0,0)=0$, whereas the effect of $v$ averaged over $w$ with equal weights is $\tfrac12[\mu(0,1)-\mu(0,0)]+\tfrac12[\mu(1,1)-\mu(1,0)]=1$. Hence the coefficient on a treatment indicator in such a model and the effect of that treatment averaged over the other factor are different quantities in general.

: SAP.P034.T3Statement

Theorem: SAP.P034.T3

Statement hash: 9977b97fd29c91518d0d840e74af8cfce26380a335ec2b889eb99e121c1d0a16

Statement definition: SAP.P034.T3Statement

Certificate. 20261103T230208Z

Declaration review: faithful; 2026-11-04T12:53:35Z; binding v2

Evidence scope: Certification checks formal proof objects. Statement fidelity is a model judgment. Author review is separate.

Proof dependencies: Independently extracted

Source and evidence metadata · sap-2026-11-08-assured-release

P034-T4

Tables OA5 and OA6, titles, first-row labels and notes, as repaired by Sol

Online Appendix p. 13 (PDF p. 13), Table OA5, and p. 14 (PDF p. 14), Table OA6

online Appendix pp. 13-14, corrected Tables OA5-OA6 labels. The fitted interaction model has the stated reference and weighted contrasts, with an average equal to its main coefficient exactly when the weighted interaction is zero, or, for positive weights, when the interaction is zero.

Historical component credits

Proved AronowAudit.P034.corrected_factorial_contrasts Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Harmless strengthening with no excluded target cases. Weights are arbitrary reals in the target already, and the positive-weight equivalences correctly handle the exceptional zero-weight case. This is fitted-function algebra, not a causal identification theorem or a verification of empirical intervals. The claim-stage verdict is unchanged.

Formal result: whole

Formal statement · Proof · Definitions

Assumptions and verification

Claim revision: catalog:3d1b895e8614de5ba6c32a9852d9d880363a17565e3567ae8e8e863c7087df95

Effective claim (source TeX): [Sol's corrected text (validated.json, P034-A001, prior_correction.corrected_text), verbatim] “Tables OA5–OA6 report reference-condition main coefficients from covariate-adjusted models including information, video type, and their interaction. OA5 compares information with no information at real video (deepfake=0); OA6 compares deepfake with real video at no information (warning=0). The displayed attrition intervals include zero. These are not effects averaged across the other treatment factor; reporting such averages would require separately specified contrasts.” [Corrected row labels, from smallest_repair] Table OA5: “Information treatment effect at real video (deepfake=0)”; Table OA6: “Deepfake treatment effect without information (warning=0)”. The 18 printed numbers are unchanged. [Mathematical content, with every hypothesis] [Setting] Let $a,b,c,d\in\mathbb R$ and $\gamma\in\mathbb R^k$ be arbitrary, and let $m(w,v,z)=a+bw+cv+dwv+\gamma'z$ for $w,v\in\{0,1\}$ and $z\in\mathbb R^k$ be the fitted regression function, where $w$ is the information indicator, $v$ the deepfake indicator and $z$ the covariate vector. Nothing is assumed about the coefficients. [Statement] For every $z\in\mathbb R^k$: (i) $m(1,v,z)-m(0,v,z)=b+dv$ for $v\in\{0,1\}$; in particular the information contrast at the real video is $m(1,0,z)-m(0,0,z)=b$ and at the deepfake video is $m(1,1,z)-m(0,1,z)=b+d$. (ii) $m(w,1,z)-m(w,0,z)=c+dw$ for $w\in\{0,1\}$; in particular the deepfake contrast without information is $m(0,1,z)-m(0,0,z)=c$ and with information is $m(1,1,z)-m(1,0,z)=c+d$. (iii) For every real weight $q$ on $v=1$ (in particular every $q\in[0,1]$), the information contrast averaged over video type is $(1-q)[m(1,0,z)-m(0,0,z)]+q[m(1,1,z)-m(0,1,z)]=b+qd$. (iv) For every real weight $p$ on $w=1$ (in particular every $p\in[0,1]$), the deepfake contrast averaged over information is $(1-p)[m(0,1,z)-m(0,0,z)]+p[m(1,1,z)-m(1,0,z)]=c+pd$. (v) $b+qd=b$ if and only if $qd=0$, and $c+pd=c$ if and only if $pd=0$; so for positive weights the coefficient equals the averaged contrast only when $d=0$. Consequently the first row of Table OA5, the coefficient $b$, is the information effect at the real video, and the first row of Table OA6, the coefficient $c$, is the deepfake effect without information; neither is an effect averaged over the other randomized factor unless the interaction coefficient is zero.

: SAP.P034.T4Statement

Theorem: SAP.P034.T4

Statement hash: bc087d25c1d897d4bbf7340ee881842db853508dbc1218b6de32129a214ddb50

Statement definition: SAP.P034.T4Statement

Certificate. 20261103T230208Z

Declaration review: faithful; 2026-11-04T12:53:35Z; binding v2

Evidence scope: Certification checks formal proof objects. Statement fidelity is a model judgment. Author review is separate.

Proof dependencies: Independently extracted

Source and evidence metadata · sap-2026-11-08-assured-release

P034-T5

Table OA3 (Study 2 - Hand-coded Romney facts across conditions), Total column, rows “Subjective Evaluation of Character” and “Stance on Other Policies”, corrected

Online Appendix p. 11 (PDF p. 11)

online Appendix p. 11, Table OA3, corrected Total column. The two corrected percentages round to 9.2 and 0.8, the two printed values fail the rounding rule, and all 39 corrected category percentages and all row and column count totals satisfy the stated arithmetic.

Historical component credits

Proved AronowAudit.P034.corrected_oa3_table Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Exact match to T5(i)–(iv), including all 39 corrected cells and all count totals. The half-open rounding interval is the target’s own convention, and total division has no effect because the denominators are positive. The claim-stage verdict is unchanged.

Formal result: whole

Formal statement · Proof · Definitions

Assumptions and verification

Claim revision: catalog:99d675fde73c4d0cd26d355c5cd738f77f7d146457e46370258634d147c9d101

Effective claim (source TeX): [Printed, Appendix p. 11, Table OA3; columns: No Info about Deepfakes | Info about Deepfakes | Total] “Subjective Evaluation of Character (e.g., trusworthy, traitor, etc.)”: 9.2% | 9.3% | 9.3%; N: 266 | 268 | 534. “Stance on Other Policies”: 0.8% | 0.9% | 0.9%; N: 24 | 25 | 49. “Total”: 100% | 100% | 100%; N: 2,898 | 2,877 | 5,775. [Corrected, from Sol's smallest_repair] In the Total column the percentage for Subjective Evaluation of Character is 9.2% (printed 9.3%) and the percentage for Stance on Other Policies is 0.8% (printed 0.9%). [Mathematical content] Each percentage in the table is $100$ times the count in its cell divided by the count at the foot of its column, rounded to one decimal place; a one-decimal number $r$ is the rounding of $x$ when $r-\tfrac1{20}\le x<r+\tfrac1{20}$. (i) $266+268=534$, $24+25=49$ and $2898+2877=5775$. (ii) $100\cdot\frac{534}{5775}=\frac{712}{77}$ and $\frac{915}{100}\le\frac{712}{77}<\frac{925}{100}$, so the rounding is $9.2$; the printed $9.3$ is not the rounding, because $\frac{712}{77}<\frac{925}{100}$. (iii) $100\cdot\frac{49}{5775}=\frac{28}{33}$ and $\frac{75}{100}\le\frac{28}{33}<\frac{85}{100}$, so the rounding is $0.8$; the printed $0.9$ is not the rounding, because $\frac{28}{33}<\frac{85}{100}$. (iv) With these two cells corrected, each of the 39 category percentages of Table OA3 (13 rows by 3 columns, transcribed in definitions_needed) is the rounding of $100\times$ count $/$ column total, each row's Total count is the sum of its two condition counts, and each column of counts sums to the total printed at its foot ($2898$, $2877$, $5775$).

: SAP.P034.T5Statement

Theorem: SAP.P034.T5

Statement hash: de07653f1e82b632f0b24e68608082bcbf4d07bcdc337e027e6d3f25f6fa219a

Statement definition: SAP.P034.T5Statement

Certificate. 20261103T230208Z

Declaration review: faithful; 2026-11-04T12:53:35Z; binding v2

Evidence scope: Certification checks formal proof objects. Statement fidelity is a model judgment. Author review is separate.

Proof dependencies: Independently extracted

Source and evidence metadata · sap-2026-11-08-assured-release

Reproduction

What I recalculated or reran from public materials. "Agrees", "Agrees in part" or "Disagrees" compares my result with the printed one. "Not compared" means the comparison could not be made.

Status: Full reproduction of the selected headline targets
Agreement: Agrees in part

Reported results that were targeted
Target	Where	Method	Status	Agreement
Figure 2, left panel: Study 1 deepfake video	p. 8 (PDF p. 8), associated text p. 7	I translated the Stata analysis into R. The effect, response shares, Cohen's d and complete-case p-value agree to rounding, and the plotted interval agrees visually. The printed $p<0.001$ differs from $p=0.00304$ under the appendix's stated worst-case attrition procedure; the effect remains significant at 5%.	Reproduced	Agrees in part
Figure 2, middle panel: Study 1 real video	p. 8 (PDF p. 8), footnote 16	I translated the Stata analysis into R. The effect of 0.726, its complete-case $p$ -value, response shares and Cohen's $d$ agree to rounding; the plotted interval agrees visually.	Reproduced	To rounding
Figure 2, right panel: Study 2 real Romney video	p. 8 (PDF p. 8), footnote 18	I translated the Stata analysis into R. The effect of 0.264, its $p$ -value, response shares and Cohen's $d$ agree to rounding; the plotted interval agrees visually.	Reproduced	To rounding
No improvement in discernment: treatment interaction	p. 8 (PDF p. 8), paragraph after Figure 2 and footnote 17	I translated the Stata analysis into R. The interaction of $-0.214$ and its $p$ -value of 0.191 agree with the printed direction and $p=0.19$ ; I did not compute a separate interaction test with attrition bounds.	Reproduced	To rounding
Pooled effects on belief and uncertainty	pp. 8-9 (PDF pp. 8-9), footnote 19	I translated the Stata analysis into R. The effects of 0.398 and 0.125 scale points and 6.0 percentage points, and their $p$ -values, agree with the printed values to rounding; these regressions omit attrition bounds.	Reproduced	To rounding
Study 2 open-ended policy associations	p. 9 (PDF p. 9), paragraph 2 and footnote 20; refers to Appendix Table OA3	I translated the Stata analysis into R and regenerated the conservative hand codes. The 3.1-percentage-point reduction in pro-choice associations and its $p$ -value agree to rounding, as does $p=0.95$ for receiving a pro-choice or pro-life code. These comparisons use the original codes before the recodes proposed in P034-E007.	Reproduced	To rounding
Study 2 closed-ended policy knowledge	p. 9 (PDF p. 9), paragraph 2	I translated the Stata analysis into R. Reversing the sign of the coded disbelief outcome gives a 14.1-percentage-point decrease in true responses; the effect and its $p$ -value agree to rounding.	Reproduced	To rounding