Anatomy of an Online Misinformation Network

Massive amounts of fake news and conspiratorial content have spread over social media before and after the 2016 US Presidential Elections despite intense fact-checking efforts.

How do the spread of misinformation and fact-checking compete? What are the structural and dynamic characteristics of the core of the misinformation diffusion network, and who are its main purveyors? How to reduce the overall amount of misinformation?

To explore these questions we built Hoaxy, an open platform that enables large-scale, systematic studies of how misinformation and fact-checking spread and compete on Twitter. Hoaxy captures public tweets that include links to articles from low-credibility and fact-checking sources. We perform k-core decomposition on a diffusion network obtained from two million retweets produced by several hundred thousand accounts over the six months before the election

As we move from the periphery to the core of the network, fact-checking nearly disappears, while social bots proliferate. The number of users in the main core reaches equilibrium around the time of the election, with limited churn and increasingly dense connections. We conclude by quantifying how effectively the network can be disrupted by penalizing the most central nodes. These findings provide a first look at the anatomy of a massive online misinformation diffusion network.

Summary

To the best of our knowledge, the following is the first in-depth analysis of the diffusion network of online misinformation and fact-checking in the period of the 2016 US Presidential Election.

The viral spread of online misinformation is emerging as a major threat to the free exchange of opinions, and consequently to democracy. Recent Pew Research Center surveys found that 63% of Americans do not trust the news coming from social media, even though an increasing majority of respondents uses social media to get the news on a regular basis (67% in 2017, up from 62% in 2016). Even more disturbing, 64% of Americans say that fake news have left them with a great deal of confusion about current events, and 23% also admit to passing on fake news stories to their social media contacts, either intentionally or unintentionally.

Misinformation is an instance of the broader issue of abuse of social media platforms, which has received a lot of attention in the recent literature. The traditional method to cope with misinformation is to fact-check claims. Even though some are pessimistic about the effectiveness of fact-checking, the evidence is still conflicting on the issue. In experimental settings, perceived social presence reduces the propensity to fact-check. An open question is whether this finding translates to the online setting, which would affect the competition between low-and high-quality information. This question is especially pressing. Even though algorithmic recommendation may promote quality under certain conditions, models and empirical data show that high-quality information does not have a significant advantage over low-quality information in online social networks.

Technology platforms, journalists, fact checkers, and policymakers are debating how to combat the threat of misinformation. A number of systems, tools, and datasets have been proposed to support research efforts about misinformation. Mitra and Gilbert, for example, proposed CREDBANK, a dataset of tweets with associated credibility annotations. Hassan et al. built a corpus of political statements worthy of fact-checking using a machine learning approach. Some systems let users visualize the spread of rumors online. The most notable are TwitterTrails and RumorLens. These systems, however, lack monitoring capabilities. The Emergent site [26] detected unverified claims on the Web, tracking whether they were subsequently verified, and how much they were shared. The approach was based on manual curation, and thus did not scale.

The development of effective countermeasures requires an accurate understanding of the problem, as well as an assessment of its magnitude. To date, the debate on these issues has been informed by limited evidence. Online social network data provides a way to investigate how human behaviors, and in particular patterns of social interaction, are influenced by newsworthy events. Studies of news consumption on Facebook reveal that users tend to confine their attention on a limited set of pages. Starbird demonstrates how alternative news sites propagate and shape narratives around mass-shooting events.

Articles in the press have been among the earliest reports to raise the issue of fake news. Many of these analyses, however, are hampered by the quality of available data—subjective, anecdotal, or narrow in scope. In comparison, the internal investigations conducted by the platforms themselves appear to be based on comprehensive disaggregated datasets, but lack transparency, owing to the two-fold risk of jeopardizing the privacy of users and of disclosing internal information that could be potentially exploited for malicious purposes. 

Ethics statement

Hoaxy collects data about public tweets obtained through the Twitter API, in full compliance with Twitter’s Terms of Service. The Indiana University Human Subjects Committee has deemed the protocol exempt from review on the basis that the data is public and poses no risk for the subjects.

 

The reviewers who performed article verification and annotated Twitter profiles were researchers in our lab who were recruited and provided informed consent via email.

Research Questions:

• RQ1: How do the spread of misinformation and fact-checking compete?
• RQ2: What are the structural and dynamic characteristics of the core of the misinformation diffusion network, and who are its main purveyors?
• RQ3: How to reduce the overall amount of misinformation?

We pose our first question (RQ1) to investigate whether those who are responsible for spreading articles are also exposed to corrections of those articles. Regretfully, only 5.8% of the tweets in our dataset share links to fact-checking content—a 1:17 ratio with misinformation tweets. We analyze the diffusion network in the run up to the election, and find a strong core-periphery structure. Fact-checking almost disappears as we move closer to the inner core of the network, but surprisingly we find that some fact-checking content is being shared even inside the main core. Unfortunately, we discover that these instances are not associated with interest in accurate information. Rather, links to Snopes or Politifact are shared either to mock said publications, or to mislead other users (e.g., by falsely claiming that the fact-checkers found a claim to be true). This finding is consistent with surveys on the trust of fact-checking organizations, which find strong polarization of opinions.

Our second question (RQ2) is about characterizing the core of the article diffusion network. We find the main core to grow in size initially and then become stable in both size and membership, while its density continues to increase. We analyze the accounts in the core of the network to identify those users who play an important role in the diffusion of misinformation. The use of Botometer, a state-of-the-art social bot detection tool, reveals a higher presence of social bots in the main core. We also consider a host of centrality measures (in-strength, out-strength, betweenness, and PageRank) to characterize and rank the accounts that belong in the main core. Each metric emphasizes different subsets of core users, but interestingly the most central nodes according to different metrics are found to be similar in their partisan slant.

Our last question (RQ3) addresses possible countermeasures. Specifically we ask what actions platforms could take to reduce the overall exposure to misinformation. Platforms have already taken some steps this direction, by prioritizing high-quality over low-quality content. Here we consider a further step by investigating whether penalizing the main purveyors of misinformation, as identified by RQ2, yields an effective mitigation strategy. We find that a simple greedy solution would reduce the overall amount of misinformation significantly.

Bot detection

Social bots play an important role in the spread of misinformation. Researchers have built supervised learning tools to detect such automated accounts with high accuracy. We leverage such a tool, called Botometer, to evaluate Twitter accounts.

Article Verification

Our analysis considers content published by a set of websites flagged as sources of misinformation by third-party journalistic and fact-checking organizations. We merged several lists of low-credibility sources compiled by such organizations. It should be noted that these lists were compiled independently of each other, and as a result they have uneven coverage. However, there is some overlap between them.

Each article was evaluated independently by two reviewers, with ties broken by a third reviewer. We applied a broadly used rubric based on seven types of misinformation: fabricated content, manipulated content, imposter content, false context, misleading content, false connection, and satire. We also added articles that could not be verified (inconclusive). Satire was not excluded because fake-news sites often label their content as satirical, and viral satire is often mistaken for real news.

We also tracked the websites of several independent fact-checking organizations: politifact.com, snopes.com, factcheck.org, badsatiretoday.com, hoax-slayer.com, opensecrets.org, and truthorfiction.com. In April 2017 we added climatefeedback.org, which does not affect the present analysis.

Technobabble Results

(Skip to Discussion Below For Leisure Read)

Low-credibility articles vs. fact-checking

If nodes contributed to articles and fact-checking in the same proportions irrespective of core number, one would expect this fraction to remain constant as peripheral nodes are removed. On the contrary, we see a drastic decrease as k increases. Fact-checking activity almost disappears as we move toward the innermost, densest portion of the network.

We can draw several insights from this analysis. The force-directed layout algorithm splits the network in two communities. There is substantial content segregation across these two communities, which we denote as the ‘fact-checkers’ and (misinformation) ‘spreaders,’ respectively. The edges across the two groups suggests some exposure of misinformation spreaders to fact-checks.

However, it is still possible to appreciate some retweeting of fact-checking content involving spreaders even in the main core. To understand in more quantitative terms the role of fact-checking in the spread of information in the core, we characterize users according to two simple metrics. Recall that in a weighted, undirected network the strength of a node is the sum of all the weights of all its incident edges, s(v) = ∑e∈v w(e). In a directed network one can likewise define the in-strength sin and the out-strength sout, by taking the sum only on the incoming and outgoing edges, respectively. We further consider edge labels and distinguish between article (sc) and fact-check (sf) strength. For each node v ∈ V let us define two ratios, the fact-checking ratio ρf and the retweet ratio ρin.

Intuitively, a user with a value of ρf close to unity is going to be a fact-checker (as opposed to misinformation spreader), whereas an account with a value of ρin close to unity is going to be a secondary spreader of information, i.e., to amplify messages through retweets rather than post original messages. We observe that for small values of k, most users fall close to the four corners of the space, meaning that they take on exactly one of the four possible combinations of roles (‘primary spreader of articles from low-credibility sources’, ‘secondary spreader of articles from fact-checking sources’, etc.).

The fact that fact-checking still spreads in the main core is a somewhat surprising observation. Therefore we search for patterns that explain how misinformation spreaders interact with fact-checking. Manual inspection of the data let us identify three key characteristics of these retweets of fact-checking content made by spreaders in the main core: (1) they link to fact-checking articles with biased, misleading wording; (2) they attack fact-checking sites; or (3) they use language that is inconsistent with the stance of a fact-checking article, for example implying that a claim is true even though the linked fact-checking article states that it is false.

Core Contributor Analysis

For at least some of the most important individuals in the network core, it would be useful to have a behavioral description at a more granular level than just group averages. To this end, we need first to define a subset of important accounts. The network science toolbox provides us with several methods to identify salient nodes.

For each measure, we rank the accounts in the main core and consider the top ten users. This exercise yields four lists of accounts. There is little or no overlap between these lists, and their union yields 34 unique accounts. Having identified a subset of moderate size that includes main core members of potential interest, we performed a qualitative analysis of these accounts. Three human raters were asked to inspect the Twitter profile of each user independently, and to provide answers to the following questions:

• Bot or Human?
• Partisanship?
• Personal or organizational account?
• How often does it share articles from low-credibility sources?

For questions 1–3, whose answer is a categorical variable, the raters could also choose ‘neither’ or ‘not sure’. After the annotators coded each account, for each question we applied a majority rule to identify the consensus label. The few cases in which a consensus could not be reached were broken by a fourth rater (one of the authors). We report results for 32 of the 34 original accounts, since two accounts had been suspended by Twitter, and thus could not be annotated. Many of the central accounts appear to be automated and display a high degree of partisanship, all in support of the same candidate.

Network robustness

Our final research question is about the overall robustness of the network. We ask: How much does the efficient spread of articles from low-credibility sources rely on the activity of the most central nodes? To explore this question we apply node disconnection, a standard procedure for estimating robustness in network science. The idea is to remove one node at a time, and analyze how two simple metrics are curtailed as a result: total volume of article retweets, and total number of unique article links. The more these quantities can be reduced by removing a small number of nodes, the more the efficiency of the misinformation network is disrupted.

We wish to measure the fraction of retweets remaining after simulating the scenario in which a certain number of accounts are disconnected, by removing all edges to and from those accounts. There is one caveat. The retweet metadata from Twitter does not allow to reconstruct the full retweet cascades. Instead, a retweet connects directly to the root of the cascade tree, so that the cascade tree is projected onto a star network. This means that when we disconnect a retweeting node, only the single leaf node is removed from the star, rather that the subtree rooted at that node in the actual cascade tree.

We prioritize accounts to disconnect based on the four centrality metrics discussed before (sin, sout, betweenness, and PageRank). The greedy strategy that ranks users by decreasing out-strength achieves the best reduction of both metrics. The efficiency of the network is greatly impaired even after disconnecting as few as 10 most influential accounts (i.e., with greatest sout). Surprisingly, disconnecting nodes with the highest sin is not as efficient a strategy for reducing misinformation; the network is robust with respect to the removal prolific accounts in the core. Betweenness, in comparison, seems to give good results on the total number of retweets, but does not produce better results than PageRank and in-strength when considering unique links (right panel).

From a policy perspective, we are not proposing that a social media platform should suspend accounts whose posts are highly retweeted. Of course, platforms must take great care in minimizing the chances that a legitimate account is suspended. However, platforms do use various signals to identify and penalize low-quality information. The present analysis suggests that the use of sout in the article spreading network might provide a useful signal to prioritize further review, with the goal of mitigating the spread of misinformation. Such an approach assumes the availability of a list of low-quality sources, which can be readily compiled.

Discussion

The rise of digital misinformation is calling into question the integrity of our information ecosystem. Here we made two contributions to the ongoing debate on how to best combat this threat. First, we presented Hoaxy, an open platform that enables large-scale, systematic studies of how misinformation and fact-checking spread and compete on Twitter. We described key aspects of its design and implementation. All Hoaxy data is available through an open API. Second, using data from Hoaxy, we presented an in-depth analysis of the misinformation diffusion network in the run up to and wake of the 2016 US Presidential Election. We found that the network is strongly segregated along the two types of information circulating in it, and that a dense, stable core emerged after the election. We characterized the main core in terms of multiple centrality measures and proposed an efficient strategies to reduce the circulation of information by penalizing key nodes in this network. The networks used in the present analysis are available on an institutional repository (see Methods).

Our analysis provides a complete picture of the anatomy of the misinformation network. Of course, our methodology has some unavoidable limitations.

First of all, Hoaxy only tracks a fixed, limited set of sources, due to data volume restrictions in the public Twitter API. Of these sources, it only tracks how their content spreads on Twitter, ignoring other social media platforms. Facebook, by far the largest social media platform, does not provide access to data on shares, ostensibly for privacy reasons, even though a significant fraction of misinformation spreads via its pages [30], which are understood to be public. Thus we acknowledge that coverage of our corpus of misinformation is incomplete. Nonetheless, by focusing on low-credibility sources that have come to the attention of large media and fact-checking organizations, and that have been flagged as the most popular purveyors of unverified claims, Hoaxy captures a broad snapshot of misinformation circulating online.

Second, Hoaxy does not track the spread of unsubstantiated claims in the professional mainstream press. News websites do report unverified claims, in a manner and with a frequency dictated by their own editorial standards. For example, hedging language is often used to express degrees of uncertainty [59]. While most claims reported in the mainstream media are eventually verified, many remain unverified, and some even turn out to be false. Some instances of misinformation may see their spread boosted as a result of additional exposure on mainstream news outlets. Understanding the dynamics of the broader media and information ecosystem is therefore needed to fully comprehend the phenomenon of digital misinformation, but it is outside the scope of the present work.

Third, we consider only US-based sources publishing English content. This is an unavoidable consequence of our reliance on lists produced by US-based media organizations. Different sources will be of course active in different countries. Worrisome amounts of misinformation, for example, have been observed in the run-up to the general elections in France [14]. To foster the study of misinformation in non-US contexts, we have released the code of Hoaxy under an open-source license, so that other groups can build upon our work [60, 61].

Last but not least, it is important to reiterate that the articles collected by Hoaxy are in general not verified. Inspection of our corpus confirms that not all articles collected by Hoaxy are completely inaccurate. As far as the present analysis is concerned, we provide an assessment of the rate of confirmed articles in our dataset (see Methods). When used as a search engine for misinformation, Hoaxy addresses this limitation by showing the most relevant fact-checking articles matching the input query, thereby facilitating claim verification. We hope that the data, software, and visualizations offered by the Hoaxy platform will be useful to researchers, reporters, policymakers, and, last but not least, ordinary Internet users as they learn to cope with online misinformation.

Citation: Shao C, Hui P-M, Wang L, Jiang X, Flammini A, Menczer F, et al. (2018) Anatomy of an online misinformation network. PLoS ONE 13(4): e0196087. https://doi.org/10.1371/journal.pone.0196087

References:

1. Barthel M, Mitchell A, Holcomb J. Many Americans Believe Fake News Is Sowing Confusion; 2016. Available from:  http://www.journalism.org/2016/12/15/many-americans-believe-fake-news-is-sowing-confusion/ .
2. Gottfried J, Shearer E. News Use Across Social Media Platforms 2017; 2017. Available from:  http://www.journalism.org/2017/09/07/news-use-across-social-media-platforms-2017/ .
3. Barthel M, Mitchell A. Americans’ Attitudes About the News Media Deeply Divided Along Partisan Lines; 2017. Available from:  http://www.journalism.org/2017/05/10/americans-attitudes-about-the-news-media-deeply-divided-along-partisan-lines/ .
4. Ratkiewicz J, Conover M, Meiss M, Gonçalves B, Patil S, Flammini A, et al. Truthy: Mapping the Spread of Astroturf in Microblog Streams. In: Proceedings of the 20th International Conference Companion on World Wide Web. WWW’11. New York, NY, USA: ACM; 2011. p. 249–252. Available from:  http://doi.acm.org/10.1145/1963192.1963301 .
5. Xiang W, Zhilin Z, Xiang Y, Yan J, Bin Z, Shasha L. Finding the hidden hands: a case study of detecting organized posters and promoters in SINA weibo. China Communications. 2015;12(11):1–13. View Article ,Google Scholar
6. Ratkiewicz J, Conover M, Meiss M, Goncalves B, Flammini A, Menczer F. Detecting and Tracking Political Abuse in Social Media. In: Proc. International AAAI Conference on Web and Social Media. Palo Alto, CA: AAAI; 2011. p. 297–304. Available from:  https://www.aaai.org/ocs/index.php/ICWSM/ICWSM11/paper/view/2850 .
7. Sampson J, Morstatter F, Wu L, Liu H. Leveraging the Implicit Structure Within Social Media for Emergent Rumor Detection. In: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. CIKM’16. New York, NY, USA: ACM; 2016. p. 2377–2382. Available from:  http://doi.acm.org/10.1145/2983323.2983697 .
8. Wu L, Morstatter F, Hu X, Liu H. Mining Misinformation in Social Media. In: Thai MT, Wu W, Xiong H, editors. Big Data in Complex and Social Networks. Business & Economics. Boca Raton, FL: CRC Press; 2016. p. 125–152.
9. Declerck T, Osenova P, Georgiev G, Lendvai P. Ontological Modelling of Rumors. In: TrandabăŢ D, Gîfu D, editors. Linguistic Linked Open Data: 12th EUROLAN 2015 Summer School and RUMOUR 2015 Workshop, Sibiu, Romania, July 13-25, 2015, Revised Selected Papers. Berlin/Heidelberg, Germany: Springer International Publishing; 2016. p. 3–17. Available from:  https://doi.org/10.1007/978-3-319-32942-0_1 .
10. Kumar S, West R, Leskovec J. Disinformation on the Web: Impact, Characteristics, and Detection of Wikipedia Hoaxes. In: Proceedings of the 25th International Conference on World Wide Web. WWW’16. Republic and Canton of Geneva, Switzerland: International World Wide Web Conferences Steering Committee; 2016. p. 591–602. Available from:  https://doi.org/10.1145/2872427.2883085 .
11. Varol O, Ferrara E, Menczer F, Flammini A. Early detection of promoted campaigns on social media. EPJ Data Science. 2017;6(1):13. View Article , Google Scholar
12. Varol O, Ferrara E, Davis CA, Menczer F, Flammini A. Online Human-Bot Interactions: Detection, Estimation, and Characterization. In: Proc. International AAAI Conference on Web and Social Media. Palo Alto, CA: AAAI; 2017. p. 280–289. Available from:  https://aaai.org/ocs/index.php/ICWSM/ICWSM17/paper/view/15587 .
13. Ferrara E, Varol O, Davis C, Menczer F, Flammini A. The Rise of Social Bots. Commun ACM. 2016;59(7):96–104. View Article , Google Scholar
14. Ferrara E. Disinformation and social bot operations in the run up to the 2017 French presidential election. First Monday. 2017;22(8). View Article , Google Scholar
15. Shao C, Ciampaglia GL, Varol O, Flammini A, Menczer F. The spread of misinformation by social bots. CoRR; 2017. arXiv:1707.07592.
16. Ecker UKH, Hogan JL, Lewandowsky S. Reminders and Repetition of Misinformation: Helping or Hindering Its Retraction? Journal of Applied Research in Memory and Cognition. 2017;6(2):185–192. View Article , Google Scholar
17. Nyhan B, Reifler J. Estimating Fact-checking’s Effects; 2016. Available from:  https://www.americanpressinstitute.org/wp-content/uploads/2016/09/Estimating-Fact-Checkings-Effect.pdf .
18. Jun Y, Meng R, Johar GV. Perceived social presence reduces fact-checking. Proceedings of the National Academy of Sciences. 2017;114(23):5976–5981.View Article , Google Scholar
19. Nematzadeh A, Ciampaglia GL, Menczer F, Flammini A. How algorithmic popularity bias hinders or promotes quality. CoRR; 2017. arXiv:1707.00574.
20. Qiu X, F M Oliveira D, Sahami Shirazi A, Flammini A, Menczer F. Limited individual attention and online virality of low-quality information. Nature Human Behavior. 2017;1:0132–. View Article , Google Scholar
21. Wardle C. Fake news. It’s complicated. First Draft News; 2017. Available from:  https://firstdraftnews.com/fake-news-complicated/ .
22. Mitra T, Gilbert E. CREDBANK: A Large-Scale Social Media Corpus With Associated Credibility Annotations. In: Proc. International AAAI Conference on Web and Social Media. Palo Alto, CA: AAAI; 2015. p. 258–267. Available from:  https://www.aaai.org/ocs/index.php/ICWSM/ICWSM15/paper/view/10582 .
23. Hassan N, Li C, Tremayne M. Detecting Check-worthy Factual Claims in Presidential Debates. In: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management. CIKM’15. New York, NY, USA: ACM; 2015. p. 1835–1838. Available from:  http://doi.acm.org/10.1145/2806416.2806652 .
24. Metaxas PT, Finn S, Mustafaraj E. Using TwitterTrails.com to Investigate Rumor Propagation. In: Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work & Social Computing. CSCW’15 Companion. New York, NY, USA: ACM; 2015. p. 69–72. Available from:  http://doi.acm.org/10.1145/2685553.2702691 .
25. Carton S, Park S, Zeffer N, Adar E, Mei Q, Resnick P. Audience Analysis for Competing Memes in Social Media. In: Proc. International AAAI Conference on Web and Social Media. Palo Alto, CA: AAAI; 2015. p. 41–50. Available from:  https://www.aaai.org/ocs/index.php/ICWSM/ICWSM15/paper/view/10592 .
26. Silverman C. Emergent; 2015. Available from:  http://www.emergent.info/ .
27. Ciampaglia GL, Mantzarlis A, Maus G, Menczer F. Research Challenges of Digital Misinformation: Toward a Trustworthy Web. AI Magazine. 2018;in press.
28. Lazer D, Baum M, Grinberg N, Friedland L, Joseph K, Hobbs W, et al. Combating Fake News: An Agenda for Research and Action; 2017. Available from:  https://shorensteincenter.org/combating-fake-news-agenda-for-research/ .
29. Lu X, Brelsford C. Network Structure and Community Evolution on Twitter: Human Behavior Change in Response to the 2011 Japanese Earthquake and Tsunami. Scientific Reports. 2014;4:6773. pmid:25346468 View Article , PubMed/NCBI , Google Scholar
30. Del Vicario M, Bessi A, Zollo F, Petroni F, Scala A, Caldarelli G, et al. The spreading of misinformation online. Proc National Academy of Sciences. 2016;113(3):554–559.View Article , Google Scholar
31. Schmidt AL, Zollo F, Del Vicario M, Bessi A, Scala A, Caldarelli G, et al. Anatomy of news consumption on Facebook. Proceedings of the National Academy of Sciences. 2017;114(12):3035–3039. View Article , Google Scholar
32. Starbird K. Examining the Alternative Media Ecosystem Through the Production of Alternative Narratives of Mass Shooting Events on Twitter. In: Proceedings of the International AAAI Conference on Web and Social Media (ICWSM); 2017. p. 230–239. Available from:  https://www.aaai.org/ocs/index.php/ICWSM/ICWSM17/paper/view/15603 .
33. Silverman C. Viral Fake Election News Outperformed Real News On Facebook In Final Months Of The US Election; 2016. Available from:  https://www.buzzfeed.com/craigsilverman/viral-fake-election-news-outperformed-real-news-on-facebook .
34. Weedon J, Nuland W, Stamos A. Information Operations and Facebook; 2017. Available from:  https://fbnewsroomus.files.wordpress.com/2017/04/facebook-and-information-operations-v1.pdf .
35. Mosseri A. News Feed FYI: Showing More Informative Links in News Feed; 2017. Available from:  https://newsroom.fb.com/news/2017/06/news-feed-fyi-showing-more-informative-links-in-news-feed/ .
36. Crowell C. Our Approach to Bots & Misinformation; 2017. Available from:  https://blog.twitter.com/official/en_us/topics/company/2017/Our-Approach-Bots-Misinformation.html .
37. Shao C, Ciampaglia GL, Flammini A, Menczer F. Hoaxy: A Platform for Tracking Online Misinformation. In: Proceedings of the 25th International Conference Companion on World Wide Web. WWW’16 Companion. Republic and Canton of Geneva, Switzerland: International World Wide Web Conferences Steering Committee; 2016. p. 745–750. Available from:  https://doi.org/10.1145/2872518.2890098 .
38. Brandtzaeg PB, Følstad A. Trust and Distrust in Online Fact-checking Services. Commun ACM. 2017;60(9):65–71.View Article , Google Scholar
39. Gomes B. Our latest quality improvements for Search; 2017. Available from:  https://blog.google/products/search/our-latest-quality-improvements-search/ .
40. Dorogovtsev SN, Goltsev AV, Mendes JFF. k-Core Organization of Complex Networks. Phys Rev Lett. 2006;96:040601 pmid:16486798 View Article , PubMed/NCBI , Google Scholar
41. Alvarez-Hamelin JI, Dall’Asta L, Barrat A, Vespignani A. K-core decomposition of Internet graphs: hierarchies, self-similarity and measurement biases. Networks and Heterogeneous Media. 2008;3(2):371–393. View Article , Google Scholar ,
42. Kitsak M, Gallos LK, Havlin S, Liljeros F, Muchnik L, Stanley HE, et al. Identification of influential spreaders in complex networks. Nature Physics. 2010;6:888–. View Article , Google Scholar
43. Conover MD, Gonçalves B, Flammini A, Menczer F. Partisan asymmetries in online political activity. EPJ Data Science. 2012;1(1):6 View Article , Google Scholar
44. Eidsaa M, Almaas E. s-core network decomposition: A generalization of k-core analysis to weighted networks. Phys Rev E. 2013;88:062819. View Article , Google Scholar
45. Davis CA, Varol O, Ferrara E, Flammini A, Menczer F. BotOrNot: A System to Evaluate Social Bots. In: Proceedings of the 25th International Conference Companion on World Wide Web. WWW’16 Companion. Republic and Canton of Geneva, Switzerland: International World Wide Web Conferences Steering Committee; 2016. p. 273–274. Available from:  https://doi.org/10.1145/2872518.2889302 .
46. Davis CA. Botometer API; 2017. Available from:  https://botometer.iuni.iu.edu/#!/api .
47. Shao C, Menczer F, Ciampaglia GL. Hoaxy FAQ; 2017. Available from:  http://hoaxy.iuni.iu.edu/faq.html .
48. Twitter. Filter realtime Tweets; 2017. Available from:  https://developer.twitter.com/en/docs/tweets/filter-realtime/overview/statuses-filter .
49. Apache Software Foundation. Apache Lucene; 2005. Available from:  http://lucene.apache.org/ .
50. Broder AZ, Glassman SC, Manasse MS, Zweig G. Syntactic clustering of the Web. Computer Networks and ISDN Systems. 1997;29(8):1157–1166. View Article , Google Scholar
51. Gupta S, Kaiser G, Neistadt D, Grimm P. DOM-based Content Extraction of HTML Documents. In: Proceedings of the 12th International Conference on World Wide Web. WWW’03. New York, NY, USA: ACM; 2003. p. 207–214. Available from:  http://doi.acm.org/10.1145/775152.775182 .
52. LLC PL. Mercury Web Parser by Postlight; 2017. Available from:  https://mercury.postlight.com/web-parser/ .
53. Lehmann J, Gonçalves B, Ramasco JJ, Cattuto C. Dynamical Classes of Collective Attention in Twitter. In: Proceedings of the 21st International Conference on World Wide Web. WWW’12. New York, NY, USA: ACM; 2012. p. 251–260. Available from:  http://doi.acm.org/10.1145/2187836.2187871 .
54. Shao C, Menczer F, Ciampaglia GL. Hoaxy API Documentation; 2017. Available from:  https://market.mashape.com/truthy/hoaxy .
55. Bessi A, Ferrara E. Social bots distort the 2016 U.S. Presidential election online discussion. First Monday. 2016;21(11). View Article , Google Scholar
56. Page L, Brin S, Motwani R, Winograd T. The PageRank citation ranking: Bringing order to the web. Stanford InfoLab; 1999.
57. Freeman LC. A Set of Measures of Centrality Based on Betweenness. Sociometry. 1977;40(1):35–41. View Article , Google Scholar
58. Callaway DS, Newman MEJ, Strogatz SH, Watts DJ. Network Robustness and Fragility: Percolation on Random Graphs. Phys Rev Lett. 2000;85:5468–5471. pmid:11136023 View Article , PubMed/NCBI , Google Scholar
59. Silverman C. Lies, Damn Lies and Viral Content: How News Websites Spread (and Debunk) Online Rumors, Unverified Claims and Misinformation. Tow Center for Digital Journalism; 2015. Available from:  https://towcenter.org/research/lies-damn-lies-and-viral-content/ .
60. Shao C, Menczer F, Ciampaglia GL. Hoaxy Backend; 2017. Available from:  https://github.com/IUNetSci/hoaxy-backend .
61. Shao C, Wang L, Serrette B, Pentchev V, Menczer F, Ciampaglia GL. Hoaxy Frontend; 2017. Available from:  https://github.com/IUNetSci/hoaxy-frontend .