How Twitter’s Birdwatch fact-checking project really works




Comment

Birdwatch, a crowdsourced fact-checking program that Twitter launched nationally on Oct. 6, has erupted with activity in the past two weeks as a small cadre of volunteer power-users works overtime to fact-check tweets about everything from candidates’ stances to the price of a Starbucks coffee to a video of a piglet rolling on a ball.

In one early example of its potential influence, a Nov. 1 White House tweet that credited President Biden with next year’s increase in Social Security benefits resulted in a Birdwatch label questioning the statement’s accuracy. The White House deleted the tweet, which one anonymous participant in the program viewed as a major triumph.

“It was right on-target,” the participant said of the user-submitted fact-checking label, according to a chat log shared with The Washington Post. “Context, neutral and apparently … quite effective.”

Birdwatch may be part of the future of content moderation under Elon Musk’s ownership of Twitter, where layoffs claimed half the staff, including an undetermined number of the people who were responsible for vetting questionable posts on the site. Musk has endorsed it, tweeting on Saturday, “Birdwatch (soon to be known as Community Notes) has incredible potential for improving information accuracy on Twitter!”

But the real battle over Birdwatch’s value is likely to take place now that the midterms polls have closed and the program’s volunteers are sifting through tweets by candidates prematurely claiming victory, alleging voter fraud or making other uncertain claims.

Birdwatch takes a Wikipedia-like approach to fact-checking that relies on a consensus of highly engaged amateurs rather than professional journalists or content moderators to decide which tweets deserve debunkings or could benefit from added context. Its supporters tout that bottom-up approach as an answer to the rising mistrust that has greeted fact-checking efforts by mainstream news outlets and social media sites.

Started as a small pilot program in January 2021, Birdwatch is now open to any Twitter user in the U.S. who signs up. Its participants are anonymous volunteers who write “notes” critiquing or explaining a tweet that are initially visible only to other Birdwatch participants.

Those notes are then flagged for other Birdwatch users to rate as helpful, somewhat helpful or not helpful, in a sort of competition to see whether any can garner a high-enough rating to be shown to all Twitter users. That’s what happened with the debunked White House tweet.

“I think Birdwatch has the potential to be a really good thing,” Renee DiResta, a research manager at the Stanford Internet Observatory who focuses on misinformation, said in a Nov. 2 tweet, adding that “the fact that it’s harder to dismiss any output as the work of Big Fact-check is a strong point in its favor. Wikipedia but for contextualization.”

The Birdwatch program is still small, with only a few dozen volunteers writing the majority of the notes on tweets in the past month, since Twitter made it available to all U.S. users. New participants in Birdwatch have to earn the privilege of writing their own notes by first submitting ratings of other notes that turn out to agree with the wider consensus. And they can lose the privilege by consistently writing notes rated not helpful by their peers.

Despite its small size, the program is influential. The tweets that earn notes include some of the most popular on the site, garnering millions of likes and hundreds of thousands of retweets. Birdwatch’s collective thoughts are able to break through filter bubbles and serve as a counterweight to the megaphones of some of the most influential users of Twitter, from Elon Musk to Biden to the group Occupy Democrats.

Twitter has said its research shows that viral tweets reach a significantly smaller audience once a Birdwatch note is appended, with users holding off on liking and sharing them.

From the start, Twitter has made the Birdwatch code and all its data open-source, inviting researchers to study it. An analysis by The Post shows that 177 of the 381 notes that have been rated highly enough to appear on the site since Oct. 26 were written by just 25 people. A total of 194 people have submitted notes that appeared on Twitter in the past month, and about 3,500 people have cast votes on the notes shown on posts from people such as Musk and Biden. (Two of the authors of this Post report have joined Birdwatch as users as part of their reporting; one has submitted two notes and rated several others to better understand the project’s workings.)

But while the data is out in the open, some of the most influential — and contentious — Birdwatch activity happens behind the scenes, on a private Discord server, which Twitter has not acknowledged publicly. (No authors of this report are members of the server.)

A Twitter representative established the Discord server for Birdwatch’s most dedicated contributors in November 2021, according to former members. The server is a place for community members to collaborate in real time on clarifications to high-profile tweets, according to the contributors who populate it.

In the year since, they have dedicated countless hours to nipping misinformation in the bud on the platform — and arguing about what exactly constitutes misinformation. Unpaid, they spend in some cases up to seven hours a day researching posts and adding notes and citations to tweets.

“We’re democratizing social media, and we’re all really passionate about this,” said Alec Palm, 20, a college student at the University of Oregon who participates in the Birdwatch program. “The past couple weeks, I’d say that a significant majority of the notes you’re seeing surface, especially ones on the White House and Elon, have come from our team,” he added, referring to the Discord crew.

The members of that core group span the ideological spectrum from liberal to right-wing and number about 20 or 30 who actively participate on a daily basis. Though they sometimes vehemently disagree on politics, seven members who spoke with The Post said there is a shared sense of mission even amid the arguments. Several spoke on the condition of anonymity, fearing harassment if their identities were revealed.

The server contains several smaller channels, including a “rapid response” channel dedicated to particularly viral tweets. Those have included the Nov. 1 White House tweet about Social Security and a Nov. 4 tweet by Musk claiming that “activist groups pressuring advertisers” had caused a “massive drop in revenue” for Twitter. A Birdwatch note on the Musk tweet, which reached the “helpful” threshold, cited a Guardian news article confirming that advertisers have paused spending amid concerns that Musk will scale back misinformation efforts on the platform.

In another channel, titled #Bad-faith-note-alert, Birdwatchers catalogue users who submit notes they believe are intentionally or blatantly wrong. Last week, a Twitter employee in the server thanked Birdwatch members for contributing to a “bad-faith contributor tracker” Google document, intended to help root out trolls.

Musk’s Oct. 27 takeover has divided the community. Several prominent Birdwatchers said they were troubled when Twitter employees, including the group’s community manager, were abruptly laid off on Friday.

“Since [Musk] took over, we don’t have a moderator anymore,” one user said. “We’re talking to a developer at Twitter who’s in there. We don’t feel like this is going to be around much longer.”

On Nov. 2, Twitter user Harrison Krank, an active member of the Birdwatch community, suggested adding Musk to the Discord server. It was immediately met with a thumbs-down emoji.

“You want someone who has spread misinformation, who wants to literally spread Russian propaganda, and who warmly welcomed back someone who spread anti-semitic death threats into this Discord?” a Birdwatch power user replied.

Another Birdwatcher added, “The less he knows about us the better.”

Still, Krank and others in the Discord server believe Birdwatch fits with Musk’s vision. “Elon is a big fan of citizen journalists and a crowdsourced fact checking program seems right up his alley,” Krank said in an Oct. 28 message. “The only reason he would shut it down would be if it became partisan.”

Palm said that while a small group of Birdwatch contributors have outsize impact on high-profile tweets, he hopes that as the program gains traction, more community members will get involved. “Yes, we’ve carried this program so far, but we don’t want to be the only ones continuing to carry it,” he said.

Musk’s layoffs affected at least some members of the Birdwatch team, said a Twitter employee familiar with the project who was not authorized to discuss personnel matters and spoke on the condition of anonymity. Musk also cut Twitter’s entire curation team, made up of professional journalists whose jobs included debunking viral falsehoods on the site, pointing users to reputable information sources and filtering out offensive or misleading trends from its “trending” sections.

But Keith Coleman, Twitter’s vice president of product in charge of Birdwatch, was spared the cuts, and Musk has amplified Coleman’s tweets about the project.

Not everyone is convinced that crowdsourcing is a viable solution to online lies and baseless conspiracy theories.

“The two aims are not compatible, fighting misinformation at scale and having bipartisan notes, and it’s kind of by design,” said Alex Mahadevan, director of MediaWise, a digital media literacy organization based at the Poynter Institute. “In this hyperpolarized world where we can’t agree on basic facts depending on your political party, the amount of accurate notes that can surface that two sides actually agree on is negligible. It will be impossible to scale this effectively.”

Palm, though an enthusiastic Birdwatch participant, said he worries Musk is placing too much weight on the program. “It feels like the entirety of the Twitter misinformation system is now reliant on Birdwatch,” he said. “Birdwatch is important but shouldn’t be the adjudicator of facts.”

The person who wrote the note on the White House’s since-deleted Social Security tweet was a longtime participant in the Birdwatch program and was among the top 30 in writing notes that were rated highly enough to be shown on the site.

Occupy Democrats, a left-leaning advocacy group, has had eight of its tweets labeled with Birdwatch notes since Oct. 6, more than any other Twitter account. Occupy Democrats co-founder Rafael Rivero told The Post the notes are “not wholly accurate” and faulted the program for not contacting the group, unlike “a fact-checker.”

The second-most fact-checked use is Musk, who has had notes shown on six of his tweets, according to Birdwatch data. Birdwatch is available only in the United States for the time being.

The White House was not alone in deleting a tweet once it received a fact-check note. Some 25 other tweets were deleted after receiving Birdwatch notes, including one by entertainer Bette Midler and several by journalists and news organizations.

Twitter’s co-founder and former chief executive, Jack Dorsey, received a fact-check on a complaint about tracking codes in Twitter URLs in which he claimed they were new. In fact, the Birdwatch note pointed out, they had been instituted during his tenure.

Birdwatch’s secret sauce lies in a sophisticated ranking system. It doesn’t simply count up the helpful and not-helpful ratings but employs an algorithm that rewards notes for being rated highly by participants who normally disagree, such as conservatives and liberals. It’s a method known as “bridging-based ranking,” which researcher Aviv Ovadya has touted as a possible antidote to social media’s polarizing tendencies.

Tom Coates, a freelance product strategist who became a Birdwatcher in May, said he has found the rating system frustrating in an era when left and right often lack a shared set of facts. He has submitted 83 notes, but only one has appeared on a tweet.

He worries that the people who will become most involved in Birdwatch as the program opens up are those seeking to sway public opinion for personal or political reasons.

“The only reason people are doing it is because either they have an altruistic commitment to truth,” he said, “or because they want their side to win.

clarification

Birdwatch is available only to users in the United States, as the story notes.