Who fact checks the fact checkers? - Snopes

The Daily Mail Snopes Story And Fact Checking The Fact Checkers

"The Daily Mail’s article makes a number of claims about the site’s principles and organization, drawing heavily from the proceedings of a contentious divorce between the site’s founders and questioning whether the site could possibly act as a trusted and neutral arbitrator of the “truth.”

When I first read through the Daily Mail article I immediately suspected the story itself must certainly be “fake news” because of how devastating the claims were and that given that Snopes.com was so heavily used by the journalistic community, if any of the claims were true, someone would have already written about them and companies like Facebook would not be partnering with them. I also noted that despite having been online for several hours, no other major mainstream news outlet had written about the story, which is typically a strong sign of a false or misleading story. Yet at the same time, the Daily Mail appeared to be sourcing its claims from a series of emails and other documents from a court case, some of which it reproduced in its article and, perhaps most strangely, neither Snopes nor its principles had issued any kind of statement through its website or social media channels disclaiming the story...

When I reached out to David Mikkelson, the founder of Snopes, for comment, I fully expected him to respond with a lengthy email in Snopes’ trademark point-by-point format, fully refuting each and every one of the claims in the Daily Mail’s article and writing the entire article off as “fake news.”

It was with incredible surprise therefore that I received David’s one-sentence response which read in its entirety “I'd be happy to speak with you, but I can only address some aspects in general because I'm precluded by the terms of a binding settlement agreement from discussing details of my divorce.”

This absolutely astounded me. Here was the one of the world’s most respected fact checking organizations, soon to be an ultimate arbitrator of “truth” on Facebook, saying that it cannot respond to a fact checking request because of a secrecy agreement.

In short, when someone attempted to fact check the fact checker, the response was the equivalent of “it's secret.”

It is impossible to understate how antithetical this is to the fact checking world, in which absolute openness and transparency are necessary prerequisites for trust. How can fact checking organizations like Snopes expect the public to place trust in them if when they themselves are called into question, their response is that they can’t respond...

When pressed about claims by the Daily Mail that at least one Snopes employee has actually run for political office and that this presents at the very least the appearance of potential bias in Snopes’ fact checks, David responded “It's pretty much a given that anyone who has ever run for (or held) a political office did so under some form of party affiliation and said something critical about their opponent(s) and/or other politicians at some point. Does that mean anyone who has ever run for office is manifestly unsuited to be associated with a fact-checking endeavor, in any capacity?”

That is actually a fascinating response to come from a fact checking organization that prides itself on its claimed neutrality. Think about it this way – what if there was a fact checking organization whose fact checkers were all drawn from the ranks of Breitbart and Infowars? Most liberals would likely dismiss such an organization as partisan and biased. Similarly, an organization whose fact checkers were all drawn from Occupy Democrats and Huffington Post might be dismissed by conservatives as partisan and biased. In fact, when I asked several colleagues for their thoughts on this issue this morning, the unanimous response back was that people with strong self-declared political leanings on either side should not be a part of a fact checking organization and all had incorrectly assumed that Snopes would have felt the same way and had a blanket policy against placing partisan individuals as fact checkers.

In fact, this is one of the reasons that fact checking organizations must be transparent and open. If an organization like Snopes feels it is ok to hire partisan employees who have run for public office on behalf of a particular political party and employ them as fact checkers where they have a high likelihood of being asked to weigh in on material aligned with or contrary to their views, how can they reasonably be expected to act as neutral arbitrators of the truth?

Put another way, some Republicans believe firmly that climate change is a falsehood and that humans are not responsible in any way for climatic change. Those in the scientific community might object to an anti-climate change Republican serving as a fact checker for climate change stories at Snopes and flagging every article about a new scientific study on climate change as fake news. Yet, we have no way of knowing the biases of the fact checkers at Snopes – we simply have to trust that the site’s views on what constitutes neutrality are the same as ours.

When I asked for comment on the specific detailed criteria Snopes uses to screen its applicants and decide who to hire as a fact checker, surprisingly David demurred, saying only that the site looks for applicants across all fields and skills. He specifically did not provide any detail of any kind regarding the screening process and how Snopes evaluates potential hires. David also did not respond to further emails asking whether, as part of the screening process, Snopes has applicants fact check a set of articles to evaluate their reasoning and research skills and to gain insight into their thinking process.

This was highly unexpected, as I had assumed that a fact checking site as reputable as Snopes would have a detailed written formal evaluation process for new fact checkers that would include having them perform a set of fact checks and include a lengthy set of interview questions designed to assess their ability to identify potential or perceived conflicts of interest and work through potential biases.

Even more strangely, despite asking in two separate emails how Snopes assesses its fact checkers and whether it performs intra- and inter-rater reliability assessments, David responded only that fact checkers work together collaboratively and did not respond to further requests for more detail and did not answer whether Snopes uses any sort of assessment scoring or ongoing testing process to assess its fact checkers.

This raises exceptionally grave concerns about the internal workings of Snopes and why it is not more forthcoming about its assessment process. Arguing that because multiple fact checkers might work on an article, reliability is not a concern, is a false argument that shows a concerning lack of understanding about reliability and accuracy. Imagine a team of 50 staunch climate deniers all working collaboratively to debunk a new scientific study showing a clear link between industrial pollution and climate change...

In a newspaper workflow, fact checking typically occurs as an editorial function, double checking what a reporter wrote. At Snopes, fact checking is the core function of an article and thus if multiple people contributed to a fact check, it is surprising that absolutely no mention is made of them, given that at a newspaper all reporters contributing to a story are listed. Not only does this rob those individuals of credit, but perhaps most critically, it makes it impossible for outside entities to audit who is contributing to what fact check and to ensure that fact checkers who self-identify as strongly supportive or against particular topics are not assigned to fact check those topics to prevent the appearance of conflicts of interest or bias.

David also did not respond to a request for comment on why Snopes fact checks rarely mention that they reached out to the authors of the article being fact checked to get their side of the story. Indeed, Journalism 101 teaches you that when you write an article presenting someone or something in a negative light, you must give them the opportunity to respond and provide their side of the story...

David did not respond to a request for comment on this or why the site does not have a dedicated appeals page for authors of stories which Snopes has labeled false to contest that label and he also did not respond to a request to provide further detail on whether Snopes has a written formal appeals process or how it handles such requests...

This is precisely the same approach used by Facebook for its former Trending Topics team and more recently its hate speech rules (the company did not respond to a request for comment).

From the outside, Silicon Valley looks like a gleaming tower of technological perfection. Yet, once the curtain is pulled back, we see that behind that shimmering façade is a warehouse of good old fashioned humans, subject to all the same biases and fallibility, but with their results now laundered through the sheen of computerized infallibility...

At the end of the day, it is clear that before we rush to place fact checking organizations like Snopes in charge of arbitrating what is “truth” on Facebook, we need to have a lot more understanding of how they function internally and much greater transparency into their work."


If you dismiss everything the Daily Mail says, then you ignore the journalism that they do that others are unwilling and/or unable to do.

In other words, even if you agree with their partisan agenda, you should have grave concerns about how Snopes does its "fact checking". Fact checkers are not neutral and they are not the gold standard. Just because Snopes says something doesn't mean it's true; in fact, as we know from many of their "fact checks", they can be very economical with the truth.

From my archives:

Who Is Fact Checking The Media Fact Checkers?: "We had firsthand experience with errant fact checks when Snopes published one in April claiming that IBD had "resuscitated" a "false" claim about 3.5 million more registered voters than eligible voters. In fact, we'd published that editorial eight months earlier — as was obvious from the time stamp on the article itself. (It went viral this spring on Facebook.) Snopes later rewrote that section of its fact check — but never acknowledged its original mistake. It also changed the ruling on the underlying claims from "false" to "mixture.""

From Muhammad Syed - "Both of the top two have put out false articles about Islam. Politifact on FGM and Snopes on slavery by Mohammed. I've reached out to both in the past with data / citations on their error with no retractions issued. Sadly, by refusing to use the truth as the ultimate benchmark they poison the discourse at large, opening a gateway to those looking to dispute reality itself."

Snopes 'Fact Checks,' Facebook Threatens Satire Site for Mocking CNN - "The Christian satire site Babylon Bee has been fact-checked by Snopes and threatened by Facebook after it published an Onion-style article about CNN about purchasing a washing machine to “spin the news.” The Babylon Bee openly advertises itself as a satire site"

Balderdash: Links - 8th September 2018 (1) - "FACT CHECK: Do Police Kill More White People Than Black People? - "CLAIM: Police shootings kill more white Americans than black Americans.
RATING: Mixture
WHAT'S TRUE: In absolute numbers, more white people than black people are killed in police shootings (because white people outnumber black people in America)...
Of course, they ignore the fact that you [also] need to adjust the killings of black people by police by the crime rate to avoid having inaccurate and misleading "analysis""

Balderdash: Links - 20th March 2018 (2) - "FACT CHECK: Was a Woman Attacked on Camera for Wearing a 'Make America Great Again' Hat in Hollywood? - "WHAT'S TRUE
YouTube star Blaire White did get into an altercation over her red "Make America Great Again" hat when she went to an anti-Trump protest on Hollywood Boulevard.
WHAT'S FALSE
Police said White and her boyfriend initiated the altercation by crossing an LAPD dividing line meant to keep opposing sides separate to prevent violence."
Comments: ""Yeah people beat the shit out of her, but hey she was asking for it by just being there. Verdict: MIXED" Wow, fuck Snopes."
"By this logic, you could say of a woman who was raped upon entering a dark alley at night while wearing provocative clothing that it was only "partly true" that she got raped. This is worse than victim blaming, it's intentionally obscuring the fact that there was a victim in the first place""

Addendum:

Snopes Is a Sneaky Liar About California’s Bill To Ban Christian LGBT Talk - "If you haven’t already lost significant respect for Snopes as an impartial fact-checker, its analysis of a bill that bans all transactions involved in stating Christian beliefs about homosexual behavior should. That bill passed 50-18 on April 19 and is being considered in the state senate. Snopes’ insistence that California Assembly Bill 2943 would not result in the Bible being banned in California is akin to Snopes calling “demonstrably and clearly false” the claim that Joseph Stalin killed everyone around him... Constitutional law attorney Jenna Ellis concludes: “A Christian bookstore could be sued for carrying a book such as Ryan T. Anderson’s latest, When Harry Became Sally, solely because the message is in conflict with the LGBT agenda…. Thus, this law is not viewpoint neutral and specifically targets psychotherapists, counselors, pastors, lay counselors, authors, speakers, and any other speakers from promoting a message of heterosexuality, and instead allows only a message affirming the LGBT viewpoint.” The Snopes articles makes it sound like Christians have nothing to fear from this bill, that the bill won’t have the effect of chilling all speech and inhibiting the sale and use of all texts that indicate that homosexual practice and transgender identity are morally wrong. Don’t you believe it for a moment."

Liberal Fact-Checker Snopes Caught Approving 'Wildly Misleading,' Anti-GOP Fake News - "Politico’s Jake Sherman slammed the image for being “more incorrect than correct.” Even writers from the left-leaning Washington Post called out the meme as inaccurate... Snopes determined that the meme, which has widely been debunked and mocked, was “True.” Not “Miscaptioned” or “Mostly True” or “Mostly False.” Nope. Snopes went with its highest rating for truthfulness for a meme that had been debunked as demonstrably wrong.Perhaps the most damning evidence against Snopes is the simple fact that the original poster of the meme has since deleted it"
On the meme claiming almost everyone who voted to repeal the Affordable Care Act got voted out

Snopes Rates Liberal Meme 'True' After Former Hillary Staffer Deletes Photo and Admits It's Inaccurate - "Business Insider reporter Joe Perticone pointed out that despite having a red X placed over her face indicated that she had been voted out of Congress, Seema Verma never actually served in Congress. Politico reporter Jake Sherman tore apart the meme stating that it was “insane fake news” and was in fact “more incorrect than correct.” Kitchel later deleted the tweet and admitted that it was inaccurate"