Corrections to: Editorial Bias in Crowd-Sourced Political Information

The problem

Study 3 added facts to the pages of 151 former senators, and 20 of the facts were removed during the follow-up: 8 of 75 positive facts and 12 of 76 negative facts. The article reports that it finds no difference between the rates at which positive and negative facts are removed ($p=0.70$ for living and $p=0.47$ for deceased former senators, and $p=0.42$ pooled). Read as a statement that the tests detected no difference, this is correct. The two passages above go further. They conclude that the bias found on the pages of sitting senators is absent elsewhere, and that Wikipedia’s culture and its policy on living persons are excluded as explanations. These conclusions need more than a test that detects no difference. They need evidence that the effect on the pages of former senators is absent, or smaller than the effect on the pages of sitting senators.

On the primary measure of the article, Study 3 gives no such evidence. The hazard ratio of removal for positive against negative facts is 0.691 for former senators (S6 Table C) and 0.629 for sitting senators (S6 Table A). The 95% interval for former senators runs from 0.28 to 1.70 and contains both 1 and 0.629. Comparing the two hazard ratios directly, I find a ratio of 1.12 with $p=0.81$. Study 3 therefore cannot distinguish the absence of bias from a relative bias as large as the one found for sitting senators.

The data do contain evidence of a narrower kind. In the first days, the difference in survival between positive and negative facts was larger on the pages of sitting senators: 0.153 against $-0.027$ at 12 hours and 0.127 against $-0.054$ at 7 days, and a test rejects equality of these survival differences at both times. A smaller absolute difference does not show that relative bias is absent, because the same relative preference can produce a smaller absolute difference on pages that are edited less often. Further, the experiments randomized the valence of a fact and not the population of pages. The pages of sitting and of former senators differ in more than political activity. These comparisons therefore do not identify political activity as the cause of the bias, and they do not exclude Wikipedia’s culture or the enforcement of its policy on living persons as explanations. The second paragraph of the Discussion allows as much when it says that editors may apply the guidelines for living persons more conservatively to sitting senators.

Evidence an exact counterexample

The evidence is a recalculation from the replication files and an exact counterexample.

The recalculation reproduces the four printed hazard ratios and standard errors of S6 Tables A and C and the three $p$-values of the main text. The figure shows the estimates with their 95% intervals. To compare the two populations, I fit a Cox model to Studies 1–4 with an interaction between valence and Study 3. The ratio between the hazard ratio for former senators and the hazard ratio for sitting senators is 1.12 ($p=0.81$), or 1.09 ($p=0.85$) when the baseline hazard is stratified by study. Treating the samples as independent and approximating the log hazard ratio standard errors by $0.318/0.691$ and $0.091/0.629$, the printed entries give

At 12 hours, the difference in survival between positive and negative facts is $23/150=0.153$ for sitting senators and $-51/1900=-0.027$ for former senators, and the gap between the two populations has $p=0.0019$. At 7 days the corresponding figures are 0.127, $-0.054$ and $p=0.0038$. When the comparison for sitting senators is restricted to cited facts, the differences are 0.140 and 0.138, with $p=0.0097$ and $p=0.0091$.

The counterexample shows that such a gap in absolute differences is compatible with one and the same relative bias. Suppose that, in each population, positive facts are removed at the constant hazard $\lambda_+=-2\log(q)/12$ per hour and negative facts at the constant hazard $\lambda_-=-3\log(q)/12$ per hour, where $q=4/5$ for sitting senators and $q=99/100$ for former senators. Then $\lambda_+/\lambda_-=2/3$ in both populations. The probability that a fact survives 12 hours is $S_+(12)=\exp(-12\lambda_+)=q^2$ for a positive fact and $S_-(12)=\exp(-12\lambda_-)=q^3$ for a negative fact, which gives $S_+(12)-S_-(12)=q^2(1-q)$.

The model illustrates the logical point and is not a fit to the data.

What survives

The bias on the pages of sitting senators stands (hazard ratio 0.629, $p=0.001$). The statement that Study 3 detected no difference also stands, and so does the sentence of the abstract that reports “no evidence of an editorial bias” for former senators, read as a statement that none was detected. The early absolute differences in survival were smaller on the pages of former senators, also when the comparison is restricted to cited facts. The findings on citations do not depend on this interpretation. What is given up is the conclusion that the bias is limited to active politicians and is not systemic to Wikipedia, and the exclusion of Wikipedia’s culture and of its policy on living persons as explanations. No data and no estimate change.

The problem

The sentence “$A$ is $x$% more likely to be removed than $B$ within a given time” compares the shares of facts removed by that time, with $B$ as the reference group. If $p_A$ and $p_B$ are the two shares, it means that $p_A=(1+x/100)\,p_B$, that is, $x=100\,(p_A/p_B-1)$. The three printed figures were computed differently. Let $a$ and $b$ be the numbers of facts removed in the two groups. The replication program evaluates $100\,|a-b|/\{(a+b)/2\}$, the difference divided by the mean of the two counts. Since the groups have equal sizes, this equals $100\,|p_A-p_B|/\{(p_A+p_B)/2\}$, a symmetric percentage difference. When $p_A>p_B>0$, the symmetric difference is smaller than the relative increase. The counts are 75 negative against 52 positive facts of 150 at 12 hours, 92 against 69 of 150 at 72 hours, and 62 uncited against 32 cited facts of 100 at four hours. The printed 36%, 29% and 64% are the symmetric differences 36.22%, 28.57% and 63.83%, rounded. The relative increases are 44.23%, 33.33% and 93.75%, and rounding does not account for the gap. The article gives no convention under which “$x$% more likely” means a symmetric difference. Someone who quotes the three figures as relative increases quotes numbers that are too small.

Evidence a recalculation from the source

The evidence is a recount from the replication data and exact arithmetic. I count a fact as removed by a horizon when its removal is recorded and its stored survival time is at most the horizon, as the program does. For negative against positive facts, Studies 1, 2 and 4 are pooled, with 150 facts in each group. For uncited against cited facts, Studies 1 and 4 are pooled, with 100 facts in each group. The counts agree with the numbers written into the program. Since the groups have equal sizes, I have $p_A/p_B=a/b$, and a direct computation gives

Evaluating the three expressions in the program, I find

Each printed integer is a rounded value from the second display, and none is a rounded value from the first. To compare the two measures in general, let $p_A>p_B>0$ and $d=p_A-p_B$, and write $R=100\,d/p_B$ and $M=200\,d/(p_A+p_B)$. Then

This implies that the symmetric difference is the smaller of the two measures. The Harvard and the Yale copies of the replication program contain the same three expressions.

What survives

Correcting only these percentages leaves the three pairs of counts, the direction of the three comparisons, the Kaplan–Meier curves, and the Cox estimates with their $p$-values (hazard ratios 1.59 and 2.33) unchanged; I have not checked the correctness of those other results here. The corrected increases are larger than the printed ones. If the survival times are rebuilt from the recorded minutes (P041-E002), converted from milliseconds to hours by division by $3{,}600{,}000$, and counted with an inclusive 12-hour cutoff, then 76 negative facts are removed within 12 hours and the first increase is $600/13=46.15\ldots$%.

The problem

In Stata, ss() returns the seconds component of a datetime and hh() returns the hour. Line 17 of the replication program therefore sets the variable hourentry to the seconds within the minute of the stored edit time. The supplements call this control the “hour of entry” (note to Table E of S6 File) and the “date and time order” of the edit. The main text speaks of the time of day (p. 3) and of the “time of entry” (p. 4). The recorded edit-time strings have zero seconds. The fitted seconds values are the remainders left by the single-precision rounding of the stored times (P041-E002). They range from $0.032$ to $59.904$, and their correlation with the fractional hour of day is $-0.03$. Every adjusted model that includes hourentry controls for a different variable from the one described. These are the two models of Table B of S6 File, the models of Table E for Studies 1, 2, 4 and the second half of Study 5, the two models of Table B of S7 File, and columns 3 and 4 of S8 File. The printed entries of these models are those of a specification that the paper does not describe. The Table E model for the first half of Study 5 omits the date and the hour, as its note says, and is not affected. The analyses of S9 and S10 Files do not use the variable; on whether a test of the “time of entry” as a moderator is reported, see P041-N005.

Evidence an exact counterexample

The evidence is an exact example from the data and a re-estimation of the affected models. For a datetime $t$ in milliseconds, the two functions are

The Study 5 edit on the page of Al Franken was recorded at 17:18 on 7 November 2014. Its stored time is $t=1{,}730{,}999{,}877{,}632$, that is, 17:17:57.632, and

The program assigns $57.632$ to an edit made in hour 17. In all 551 rows the value of ss(time_edit) differs from the hour.

I re-estimated the Cox models in R with Breslow ties and robust standard errors scaled as in Stata. With the seconds as the control, I obtain the printed entries of Table B of S6 File, $0.670$ $(0.095)$ and $0.352$ $(0.055)$. The other hazard ratios shown also agree at three decimals, one entry of S7 Table B aside (printed $0.386$, re-estimated $0.389$). I then replaced the control by the integer hour of day, rebuilt from the strings that record the edit times. The table lists selected changes, and the figure shows all 20. Every hazard ratio keeps its side of 1, and every conclusion at the 5% level that I checked is unchanged. The $p$-value of “Positive” in column 2 of S7 Table B rises from $0.095$ to $0.214$. Applying hh() to the stored times is not enough, since rounding moves three edits across an hour boundary. With a fractional hour (hour plus minutes over 60), the two pooled entries are $0.691$ and $0.344$, nearly the same as with the integer hour.

What survives

The unadjusted estimates of Table A of S6 File, and with them the hazard ratios 1.59 and 2.33 of the main text, do not use the variable. The same holds for the Kaplan–Meier curves, the balance tables, Study 3, the unadjusted interactions, the regressions of S11 File and the tables of S9 and S10 Files. In the affected models the pooled effects of positive and cited facts stay significant, and no sign changes. All 20 hazard ratios of those models change at three decimals. For example, the Study 4 hazard ratio for positive facts in Table E moves from $0.75$ to $0.93$. The interaction of citation with an upcoming election, which the main text quotes as $0.32$ (p. 5), becomes $0.28$, and “Positive” in column 2 of S7 Table B is no longer significant at the 10% level.

The problem

The article’s definition of the survival time is correct; the stored values are not exact. The data file holds the three variables as Stata floats, which have 24 significant binary digits. A Stata datetime counts milliseconds from 1 January 1960, and the 2014 values lie between $2^{40}$ and $2^{41}$, where consecutive floats are $2^{17}=131{,}072$ milliseconds apart. The edit and removal times were recorded to the minute (the string variables time and removed). Each stored time is the recorded minute rounded to the nearest multiple of $131{,}072$ ms, and time_alive is the difference of the two rounded times. Of the 294 survival times, 286 differ from the difference of the recorded minutes, by between $-126{,}752$ and $+128{,}960$ ms, and 15 stored survival times are zero where the recorded minutes give 8. The multiplier $2.77778\times10^{-7}$ in line 15 is a rounded value of $1/3{,}600{,}000$ and moves a survival time of exactly 12 hours above 12.

Evidence an exact counterexample

The evidence is an exact counterexample from the data, with a recalculation. Set $q=2^{17}=131{,}072$. For the fact on the page of Jon Tester in Study 5, the recorded times are 18:49 and 20:47 on 6 November 2014, which are 118 minutes apart. In milliseconds the two times are

Since $64{,}864<q/2<66{,}976$, the first is rounded down and the second up. The stored times are $13{,}205{,}863\,q$ and $13{,}205{,}918\,q$, and I obtain

an error of $128{,}960$ ms, or $128.96$ seconds. For the negative, cited fact on the page of Dan Coats in Study 1, the recorded times are 21:23 on 9 July and 09:23 on 10 July 2014, exactly 12 hours apart, and

Both are rounded down, so that $\texttt{time\_alive}=330\,q=43{,}253{,}760$ ms, which exceeds $43{,}200{,}000$ ms. The program counts a removal within 12 hours when hours_alive is at most 12, and this fact is not counted. With the recorded minutes it is counted. The number of negative facts removed within 12 hours in Studies 1, 2 and 4 is then 76, against 75 from the stored values, and the number of positive facts stays at 52. Among the horizons of 4, 6, 12, 24, 72 and 168 hours, the only other change is one Study 5 fact at 4 hours, outside the pooled samples. The multiplier alone would also misplace the Coats fact, since $43{,}200{,}000\times2.77778\times10^{-7}=12.0000096$.

I re-estimated the two pooled Cox models of Table A of S6 File with survival times rebuilt from the recorded minutes and nothing else changed (Breslow ties, robust standard errors). The hazard ratio for positive facts moves from $0.628942$ to $0.627865$ ($p=0.0013$) and that for cited facts from $0.429227$ to $0.427530$ ($p<0.001$). Their reciprocals are $1.593$ and $2.339$, against the printed 1.59 and 2.33. In the 12-hour regressions of the kind reported in S11 File, the coefficient for positive facts moves from $0.153$ to $0.160$ and that for cited facts from $0.37$ to $0.36$, and both $p$-values stay below $0.01$.

What survives

The designs, the treatment assignments and the removal indicators are unchanged. The hazard of removal remains lower for positive facts and for cited facts, and both pooled effects remain statistically significant. The 3-day counts (69 and 92) and the 4-hour citation counts (32 and 62) are unchanged, and no fact changes its classification at 6, 24 or 168 hours. Every analysis of the survival times inherits the rounding, and its entries can change in the last digits; I recomputed only the quantities listed above. The seconds within each minute were never recorded, and the correction does not restore them.

The problem

The codebook defines lengthofincumbency as the year in which the senator was first elected, and its values in the data are calendar years, 1993 and 2013 for example. The do-file builds a tenure, yearsinsenate, as 2014 minus that year, but uses it only in the balance checks. The Cox commands behind S9 and S10 Files use the year. A row labelled length of incumbency is read per additional year of tenure. In S9 File the entry is read as the factor by which the hazard ratio of the treatment changes, and in S10 File as the hazard ratio itself. Each of the five printed entries is the corresponding factor per later year of first election, and the factor per year of tenure is its reciprocal. The direction is therefore read backwards in all three rows, among them the one starred row, 1.056***. The entries are hazard ratios, as in the other Cox tables.

Evidence a recalculation from the source

The evidence is a recalculation from the replication data and a derivation. Refitting the five models with the year of first election reproduces all five printed pairs of estimate and standard error to three decimals. Let $Y$ be the year of first election, $L=2014-Y$ the tenure, $T$ the treatment indicator (positive or cited) and $z$ the remaining terms of the linear predictor. I write the fitted hazard as $h(t\mid T,Y,z)=h_0(t)\exp(bY+aT+gTY+z)$. Substituting $Y=2014-L$, I obtain

The treatment hazard ratio at tenure $L$ is $\exp(a+2014\,g-gL)$, so one more year of tenure multiplies it by $e^{-g}=1/e^{g}$. S10 File has no treatment term, and there the hazard ratio per year of tenure is $e^{-b}$. The five models refitted with $L$ give the reciprocals in the table.

What survives

The fitted hazards, the partial likelihood, the test of each incumbency interaction, the test of the incumbency main effect and the joint moderation tests are the same under either coding. The interaction in Table B of S9 File stays significant, with $p=0.004$ and $p=0.003$ in its two columns, in the opposite direction. The main text draws no conclusion about the direction of an incumbency effect. The comparisons of treatments, the balance tables and the other coefficients of the models that use the year only as an additive control are unaffected.

The problem

The Study 3 sample has 151 former senators, 72 living and 79 deceased. The last year of Senate service is before 1990 for 75 of them, 20 living and 55 deceased. A living senator who left before 1990 fits neither “passed away” nor “left the Senate since 1990”; examples are Bill Brock (1971–1977), Dan Quayle (1981–1989) and Paul S. Trible, Jr. (1983–1989). S3 File reports the broader eligibility rule: “senators who have passed away since 1990 or are living and retired during or before 2009”, of whom 243 qualified and 151 had both facts collected. The living senators left the Senate between 1971 and 2009, with median 1997, so that “recently retired” does not convey the full range of departure dates. I did not check the clause “do not currently hold any elective office” against dated records of office, and it is neither confirmed nor refuted here.

Evidence a recalculation from the source

The evidence is a recalculation from the replication data. I take the last year in the Senate to be the last four characters of the variable term, as the replication program does; it equals the latest year among the terms listed for each senator. With this definition, I find that the means of the last year are $29831/15\approx1988.7$ for the 75 positive facts and $151157/76\approx1988.9$ for the 76 negative facts. These are the two values printed in Table A of S3 File. All 72 living senators left by 2009, in agreement with the rule of S3 File.

What survives

No estimate changes. The Study 3 curves in Fig 1 and Table C of S6 File ($N=151$, 72 and 79) describe the sample that was analyzed, and the pooled analyses of Studies 1, 2 and 4 do not use Study 3. What changes is the description of the comparison group: it is the former senators of the S3 rule, and a comparison restricted to departures since 1990 (52 of the 72 living senators) would be a separate analysis.

The problem

The supplements state three exceptions that the summary omits. S2 File says that the IP address was “only introduced beginning in Study 2”. S1 File says that after the first day of Study 1, “All subsequent edits were made from the same account, veto118”, an account that “was randomly assigned to be the first account”. S4 and S5 Files say that the ten locked pages were edited from accounts that had “already been used to make edits in previous rounds of this experiment”. Note that reusing accounts does not contradict the statement that new accounts were created.

Evidence a check of the printed page

The evidence is a check of the page against the text of S1, S2, S4 and S5 Files.

What survives

The supplements disclose the exceptions, and no estimate depends on the summary. The assignment of valence and citation, the survival times and every reported estimate are unchanged. The wording alone does not show that any estimate is biased.

The problem

The rows “N” and “Studies” describe the sample of each regression, and neither column is described correctly. The commands in the replication files restrict the first regression to Studies 1, 2 and 4 (300 edits) and the second to Studies 1, 2, 4 and all of Study 5 (400 edits). The second entry of the row “Studies” is also inconsistent with its own “N”, since Studies 1, 2, 4 and the first half of Study 5 contain 350 edits.

Evidence a recalculation from the source

The evidence is a recalculation from the replication files. On 300 and 400 edits, the fits reproduce all sixteen printed hazard ratios, all sixteen standard errors and both joint $p$-values to three decimals. On the 200 and 250 edits in which citation was randomized (Studies 1 and 4, and these two with the first half of Study 5), the same model has joint $p$-values 0.527 and 0.311. The interaction with influence then loses its marker, and the interaction with length of incumbency is significant at 5% and not at 1%.

What survives

If the two rows are corrected, the estimates, standard errors and tests in the table are unchanged. The main text (p. 5) cites the table for the statement that there is no systematic evidence that other variables moderate the effect of citations, and this statement is not weakened under either repair.

The problem

In the S8 File, columns 3 and 4 are marked “Covariates Yes”, and the note says that this includes the influence variable. The commands in the replication files for these two columns do not contain that variable, and the printed entries reproduce without it and not with it. In S9 Table A, the note repeats this definition, and the row “Covariates” marks the first column “Yes” and the second “No”. The two commands have the same terms: the interactions of “Positive” with the moderators, the main effects of the moderators and indicators for study wave. Neither command adds the date of entry, the hour of entry or Senate class, and the columns differ only in the studies included. The note to S6 Table D defines a marker of four stars that no entry of that table carries. Since Table D is labelled as unadjusted and its command agrees with the label, this definition is superfluous and not false.

Evidence a recalculation from the source

The evidence is a recalculation from the replication files. The corrected descriptions of the entry-time and incumbency variables follow P041-E003 and P041-E004. Without the influence variable, the two adjusted models of the S8 File reproduce the six printed hazard ratios 0.574, 1.557, 0.783 and 0.507, 0.323, 2.146. With it, I obtain 0.580, 1.538, 0.774 and 0.515, 0.294, 2.237. The two specifications of S9 Table A, fitted to 300 and 400 edits, reproduce all sixteen printed hazard ratios of the interactions.

What survives

The reproduced hazard ratios are unchanged, and only the notes and the flag change. The interaction between citation and an upcoming election that the main text reports as 0.32 (p. 5) is the entry 0.323 of the S8 File, fitted without the influence variable; with that variable the interaction is 0.294 with $p<0.001$.

The problem

The two tables are titled “OLS Estimates on Survival”. They regress an indicator that the fact was still present after 6, 12 or 24 hours or 7 days, and the printed slopes have the signs of that outcome. The sentence calls the outcome removal, for which every slope would have the opposite sign. The paragraph opens by naming survival, and the sentence that follows interprets the slopes correctly. The intended meaning can be recovered from them. The Harvard Dataverse copy (version 1.0) of the do-file codes the four outcomes as survival. The Yale Dataverse copy (version 2.0) codes them as removal by the stated time, and returns every slope with the sign opposite to the printed one.

Evidence a recalculation from the source

The evidence is an exact recalculation from the replication data. Let $H_i$ be the duration of fact $i$ in hours and $S_i(h)=\mathbf{1}\{H_i>h\}$. With the outcome $S_i(h)$, all twelve printed slopes of the treatment indicators are reproduced to three decimals. For example, the six-hour slope for positive facts in Studies 1, 2 and 4 is $17/150$, printed as 0.113. The removal indicator is $R_i(h)=1-S_i(h)$. Since each regression has an intercept, I have

for the two vectors of coefficients, where $e_0$ is the coordinate vector of the intercept. Every slope changes sign ($-17/150$ in the example), the residuals change sign, and the robust covariance matrix is the same.

What survives

All twelve printed slopes and their interpretation stand, and no coefficient changes. The Cox models, the Kaplan–Meier curves and Fig 1 use the duration and the failure indicator, not these four outcomes. The Yale copy of the do-file reproduces the printed signs once the coding of lines 147–154 of the Harvard copy is restored. That coding sets each of hour6, hour12, hour24 and day7 to 1, and replaces it by 0 when hours_alive is at or below the stated time.

The problem

The printed standard error is about ten times too large. A coefficient of 0.320 with a standard error of 0.645 has $t=64/129\approx0.496$ and a two-sided $p$ of about 0.62, which contradicts the three stars printed beside the coefficient. The standard errors of the same coefficient at 12 hours, 24 hours and 7 days are printed as 0.065, 0.062 and 0.053.

Evidence a recalculation from the source

The evidence is a recalculation from the replication data in exact arithmetic. I regress the indicator of survival beyond six hours on an intercept, the citation indicator and an indicator for Study 4, in Studies 1 and 4. The regression has $n=200$ and $k=3$, and its citation coefficient is $8/25=0.320$. Each of the four cells of study by citation holds 50 facts, and the sum of squared residuals is $2047/50$. The HC1 variance of the citation coefficient is

and taking the square root, I find the standard error $\sqrt{2047/492500}=0.0644697\ldots$, which rounds to 0.064. The $t$ statistic is then 4.96, with $p\approx1.5\times10^{-6}$ on 197 degrees of freedom, in agreement with the three stars.

What survives

The coefficient 0.320 and its three stars stand. The recalculated standard errors at 12 hours, 24 hours and 7 days match their printed cells, and no other calculation uses this cell.

The problem

The tables print hazard ratios with standard errors on the scale of the hazard ratio, and their notes assign one star to the 10% level, two to the 5% level and three to the 1% level. For the four entries above, the printed marker is not the one that the printed hazard ratio and standard error imply. Three markers understate the level of significance. The fourth, in S9 Table A, claims significance at 1% for an interaction that is significant at 5%.

Evidence a recalculation from the source

The evidence is a calculation from the printed numbers and a recalculation from the replication files. For a printed hazard ratio $\widehat{H}$ with standard error $s$, the Wald statistic for the log hazard ratio and its two-sided $p$-value are

where $\Phi$ is the standard normal distribution function. The table below lists $z$ and $p$ at the printed values, bounds on $p$ over all hazard ratios and standard errors that round to the printed values, and the $p$-value when the same model is fitted to the replication files. In every row these bounds lie on one side of the thresholds 0.01 and 0.05, so that rounding cannot account for the printed marker.

What survives

The four estimates and their standard errors are unchanged. The joint tests 0.321 and 0.221 of S9 Table A are also unchanged. The correction leaves in place the statement of the main text that there is no systematic evidence of moderation by covariates. The interaction with the Northeast region remains significant at 5%.

The problem

S12 File counts the 49 quality-control items among the facts that a worker could be shown: facts were sampled “from the set of 551 facts included in the five studies, plus an additional 49 facts that were seeded as quality control”, up to 50 per worker. The program drops the quality-control items in line 49 and ranks the submissions of each worker afterwards, in lines 51–53. The variable view_first therefore marks each worker’s first submitted substantive fact. Seven of the 113 workers had completed a quality-control item before that fact. For these seven the selected fact is not the first that the worker saw, and the restriction does not exclude earlier exposure to an item, which is the purpose stated for it.

Evidence an exact counterexample

The evidence is an exact counterexample from the replication data, with a recalculation. One worker submitted a quality-control item at 01:51:28 GMT on 9 December 2014 and accepted the selected fact, a Study 4 fact about Ed Markey, at 01:51:35, seven seconds later. For the seven workers the gaps between the submission of the quality-control item and the acceptance of the selected fact are 7, 3, 1, 1, 13, 6620 and 30 seconds. All six printed regressions of Table A reproduce at the printed precision. The slope of a regression on a binary indicator is the difference of the two group means. Without the seven workers, I find that the three slopes are $1535/459$, $-7/351$ and $2987/1404$.

The times are those of accepted and submitted assignments and do not show what a worker viewed. One retained worker accepted four substantive facts in the same second, so the order of viewing cannot be established for every worker.

What survives

The six printed regressions are correct for the samples that the program selects. Without the seven workers the pattern is the same: large positive differences for the positive-negative question and for favorability, and a difference near zero for relevance. The columns for all responses and the Wikipedia analyses do not use this restriction.

The problem

Each figure of the S12 File is an image embedded in the Word document, and the image stores its labels as text. In all three figures this text is malformed. The commands in the replication files that draw the figures contain the intended labels: “Negative” and “Positive” for the two groups, and tick labels such as “Very Neg” and “Somewhat Pos”.

Evidence a check of the printed page

The evidence is a check of the source file. I read the text that is stored in the three images embedded in the file, which consists of ordinary characters in the font Calibri. I did not check how the labels appear in Word with the original fonts.

What survives

No number depends on the labels, and the two groups and the response scale keep their intended meaning.

The problem

The caption and S8 File contain misprints. The sentence of S3 File concerns the accounts created to reach autoconfirmed status and itself continues “using these accounts”. S2 File describes the same step as creating five Wikipedia accounts. The pre-election study is Study 4 (30–31 October 2014), and Study 5 followed the election (6–7 November 2014). The variable pagechar is the length of the page in characters, as p. 4 and the notes to the supplementary tables say.

Evidence a recalculation from the source

The evidence is a check of the page, the supplements and the codebook, and a recalculation from the replication data. In the data, treatment_pre_order is missing in Studies 1 to 3, is filled for all 100 Study 4 records, and is copied in the same order into the matching 100 Study 5 records. The values of pagechar range from 26686 to 631299.

What survives

These five wording changes do not alter any number or fitted result.

The problem

After each panel is plotted, a loop extends every curve to the display limit of 250 hours. The extension is meant for a curve whose fit stops before 250 hours, but the loop has no such condition. It uses the final fitted survival level, which need not be the level at 250 hours. For the eight curves of Fig 1 the fit ends at 3463.58037 hours with survival zero. This is the time that the script assigns to facts that were never removed (P041-N002). The loop draws a segment at survival zero from 3463.58037 hours back to 250 hours. With R’s default axis style, xaxs="r", the horizontal range 0 to 250 is widened to $-10$ to 260, and the part of the segment between 250 and 260 hours appears inside each panel. The published Fig 1 stops at 250 hours and shows no such segment.

Evidence a recalculation from the source

The evidence is a recalculation from the replication files and a check of the page. I ran the supplied plotting code with R’s default axes and recorded the coordinates of each extension. For all eight curves the extension runs from 3463.58037 hours to 250 hours at survival zero. The render shows a zero-level segment to the right of 250 hours in each panel. A second render, with the extension drawn only when max(temp$time) < maxtime and with xaxs="i", shows none. On the published page, viewed at 220 dpi, there is no mark at survival zero at the right-hand edge of any panel.

What survives

The published Fig 1 and the survival curves it shows over 0 to 250 hours are unaffected. Only a new render of the supplied script with default axes shows the extra segments. The same loop, without a condition, draws Figure A of S6 File.

The problem

In R’s survival package, Surv called with a time and no event indicator gives every row status 1, an observed event. The outcome of the paper is the time until a fact is removed. In the data, 257 of the 551 facts were never removed: their time_alive is missing and removed is coded “NO”. Line 10 assigns these facts twice the largest recorded duration, that is, 3463.58 hours, and line 14 enters them as removals at that time. The fitted curve of any group that contains such facts then falls to zero at 3463.58 hours, where a Kaplan–Meier curve with these facts censored keeps its last level. The column dead is the missing-time indicator, not the removal indicator. It is TRUE for the facts that were never removed, and passing it as event would invert every status.

Evidence an exact counterexample

The evidence is an exact counterexample and a recalculation from the replication data. I take two records, one removed at raw time 3600000 and one never removed. The script assigns the second record the raw time 7200000, and with the conversion constant $2.77778\times10^{-7}=138889/500000000000$ the two times are $t_1=1250001/1250000$ and $t_2=2t_1$ hours. The Kaplan–Meier estimator is

where $n(u)$ is the number of records with time at least $u$ and $d(u)$ is the number of events at $u$. At $t_1$ I have $n=2$ and $d=1$ whatever the status of the second record, which gives $\widehat S(t_1)=1/2$. With the statuses $(1,1)$ of the script, $n(t_2)=d(t_2)=1$ and $\widehat S(t_2)=0$. With the statuses $(1,0)$, no factor is added at $t_2$ and $\widehat S(t_2)=1/2$.

In the data, the largest recorded duration is 6234439680 in raw units. The invented time is then $169120135296/48828125=3463.58037086208$ hours, and all 551 statuses are 1. A fact that was never removed has time $3463.58>250$ in the survival object of the script and in the repaired one, and every recorded removal is the same in both. For every $u\le250$ the two objects then have the same $n(u)$ and $d(u)$. This implies that the Kaplan–Meier products agree on $[0,250]$. In exact arithmetic on the data, the two products agree for all 26 fitted curves, 8 in Fig 1 and 18 in Figure A of S6 File. The table lists the curves of Fig 1.

What survives

Every Kaplan–Meier value shown in Fig 1 and in Figure A of S6 File, which stop at 250 hours, is unchanged when only the event indicator is repaired. The Cox regressions, the regression tables and the counts at fixed times come from the Stata do-file and do not use the R object. The do-file sets dead to 1 for a recorded removal and declares it with stset hours_alive, failure(dead). The fitted curves beyond 250 hours and any summary of the full fit do change. The repair keeps the time that the script assigns to facts that were never removed. Whether that time matches the actual end of monitoring is a separate question, which this correction does not settle.

The problem

Table A of S9 File reports interactions of the indicator for positive facts with party, influence, three regions, length of incumbency, log page length and log state population. It has no interaction with Senate class, date of entry or time of entry. S10 File regresses survival on covariates alone, with no valence indicator and no interaction, and therefore cannot test moderation. The one reported test of Senate class as a moderator of the valence effect is in S8 File, which the sentence does not cite. There Senate class is dichotomized as facing reelection in 2014, and the interaction is 1.750* (0.550) without covariates and 1.557 (0.485) with them. No supplement and neither copy of the do-file reports an interaction of valence with date or time of entry. These two variables enter only as additive controls.

Evidence a check of the printed page

The evidence is a check of the page against the twelve supplements and the replication do-files. I write the Cox model with interactions as $h(t\mid V,Z)=h_0(t)\exp\{\beta V+\gamma^\top Z+\delta^\top(VZ)\}$, where $V=1$ for a positive fact and $Z$ is the vector of covariates. Dividing the hazard of a positive fact by the hazard of a negative fact at $Z=z$, I find the hazard ratio $\exp(\beta+\delta^\top z)$. Moderation is a statement about $\delta$, and S10 File estimates only $\gamma$. The joint Wald tests of Table A in S9 File have $p=0.321$ and $p=0.221$. From the rounded entries of S8 File, the interaction with Senate class has $p\approx0.075$ without covariates and $p\approx0.155$ with them.

What survives

The analyses that are reported do not contradict the statement of no systematic evidence. The joint tests do not reject, and the interaction with Senate class is marginal only without covariates. The cited files do not document three of the eight listed moderators, and the corrected sentence covers six. The pooled valence estimates, the survival curves and the citation results do not depend on this sentence. I could not compare the sentence with the pre-analysis plan, which was not available.

The problem

In each supplement, “this” refers to a change made after editors reacted to the uncited edits: the remaining edits of Study 1 were made faster, and the remaining uncited edits of Study 5 were made cited. Random order balances, in expectation, any pattern over the calendar that does not depend on the treatments. It does not cover the case that the supplements describe, in which the treatment of earlier edits (an uncited fact from the same account) changes how later edits are received or when they are made. In that case the comparison under the protocol that was carried out need not equal the comparison under the planned protocol, or the effect of an edit’s own treatment. Random order gives no guarantee here. To keep the assurance, one needs a further assumption that excludes such effects of earlier edits on later ones. An example is the assumption that the removal of an edit does not depend on the treatments and timing of the earlier edits from the account. The supplements state no such assumption, and the replication data do not identify the editors who removed the facts. This is a limit on the assurance; it does not show that any estimate is biased.

Evidence an exact counterexample

The evidence is an exact counterexample to the sufficiency of random order, with a recalculation of two sensitivity checks. I take four pages, two assigned a cited fact (C) and two an uncited fact (U), as in one block of Study 1, and let the 24 orders of insertion be equally likely. Suppose that an uncited edit starts a period in which every later insertion from the account is removed, whatever its own citation status, and that no other removals occur. This response is hypothetical and is not a description of the actual editors. On the planned, slow schedule the period ends before the next insertion, no fact is removed, and the difference $D$ between the removal rates of cited and uncited facts is $0$. On the accelerated schedule, all remaining insertions fall inside the period that the first uncited edit starts. The edit at rank $r$ is then removed if and only if an earlier edit is uncited,

where $z_s$ is the type of the edit at rank $s$. The outcome $Y_r$ does not depend on $z_r$, which implies that the effect of an edit’s own citation is zero. The six orders of types are equally likely.

Averaging over the six orders, I obtain

against $0$ on the slow schedule, although the order is uniformly random and the citation has no effect of its own. With 100 pages, 50 cited and 50 uncited, the same construction has expected removal rates $50/51$ and $49/50$, and I find that $\mathbb{E}[D]=1/2550$. This value shows that the expectation need not vanish. It is not an estimate of any bias in the paper. If the pattern of removals is fixed over the calendar and does not depend on the treatments, the same enumeration gives $\mathbb{E}[D]=0$.

The replication data show removals concentrated in time: in Study 1, 55 of the 93 removals fall between 19:28 and 20:39 on 15 July 2014, and all 36 facts inserted on that day were removed on that day. For the pooled comparison of positive and negative facts (Studies 1, 2 and 4), I re-estimated the Cox model in R. With robust standard errors computed without clustering, as in the replication program, the hazard ratio is $0.629$ and $p=0.0013$. With standard errors clustered by senator, I find $p=0.0008$. Without the 36 Study 1 edits of the last day, the hazard ratio is $0.606$ and $p=0.0016$.

What survives

The counts, the survival curves and the fitted coefficients are unchanged, and no numerical correction follows. The pooled difference between positive and negative facts remains under clustering by senator and without the last day of Study 1. These checks do not address an effect of earlier edits on later ones within an account. Study 1 enters both pooled analyses of the main text. Study 5 enters supplementary analyses and is excluded from the main-text pooled estimates. What is withdrawn is the unconditional assurance for Studies 1 and 5.

The problem

The sentence summarizes the six kappas that follow it. A Fleiss kappa measures how far the raters agree on the exact category beyond the agreement expected from the pooled frequencies of the categories. For relevance the two values are $0.036$ and $0.077$: the raters agree in $25.2\%$ of pairs against a benchmark of $22.5\%$ with seven categories, and in $58.2\%$ against $54.7\%$ with three. As a summary that includes relevance, “fairly strong” overstates these values. The paper sets no threshold for the phrase, and the finding concerns the wording; the six numbers are correct.

Evidence a recalculation from the source

The evidence is a recalculation from the replication data, in exact arithmetic. There are 551 facts with five ratings each, and a missing rating is set to the midpoint 4, as in the replication program. Let $P_o$ be the share of agreeing pairs of raters among the $5510$ pairs and $P_e=\sum_j p_j^2$, where $p_j$ is the pooled share of category $j$. The kappa is $\kappa=(P_o-P_e)/(1-P_e)$. For relevance with seven categories I obtain

and with three categories

The six kappas are $0.202$ and $0.559$ for the positive-negative question, $0.036$ and $0.077$ for relevance, and $0.134$ and $0.381$ for favorability, and each rounds to the printed value.

What survives

The six kappas stand. The mean ratings separate the two kinds of facts: positive minus negative is $2.96$ points for the positive-negative question and $2.30$ for favorability, and $-0.03$ for relevance. The description of the facts as positive or negative rests on these means, and the analyses of survival do not use the ratings.

Intended meaning

The phrase “at statistically significant levels” applies to the two differences that Table A shows to be significant, $2.96$ (standard error $0.08$) for the positive-negative question and $2.30$ $(0.09)$ for favorability. For relevance the table shows $-0.03$ $(0.07)$, and the two mean ratings are $5.00$ and $5.03$, so “equally relevant” means that no difference in mean relevance was detected. Read in this way the sentence is correct. A test of a zero difference that does not reject is not a test of equivalence, and S12 File reports no equivalence margin or test.

Intended meaning

The entries are exponentiated Cox coefficients, that is, removal hazard ratios for the named condition against its reference, and the standard errors are on the same scale. For the row “Positive” of Table A in S6 File, the fitted coefficient on the log-hazard scale is $-0.4637$. Its exponential is 0.629, and 0.629 times the robust standard error of the coefficient is 0.091. The main text prints the reciprocals: $1.59\approx1/0.629$ on p. 4 for negative against positive facts, and $2.33\approx1/0.429$ on p. 5 for uncited against cited facts. The supplements do not use the words “hazard ratio”, but the surrounding text and these reciprocals identify the intended scale.