The Quest to Automate Fact-checking

Transcription

1 The Quest t Autmate Fact-checking Chengkai Li University f Texas at Arlingtn MetrCn, Octber 26,

2 Innvative Database and Infrmatin Systems Research Labratry Chengkai Li, Assciate Prfessr and Assciate Chair idir.uta.edu 2 Research areas Big Data Intelligence and Data Science (Database, Data Mining, Natural Language Prcessing) Research directins cmputatinal jurnalism, cmputatinal fact-checking crwdsurcing and human cmputatin cybersecurity, scial bts data explratin by ranking/skyline/preference queries graph database usability, knwledge graphs

3 Pliticians Make False and Misleading Claims Fact-checks n majr party presidential nminees. Surce: PlitiFact.cm 3

4 Natinal Security Threats Fake-news flds scial media ( filter bubbles and ech chambers ). Facebk trending tpic algrithms prmted fake-news. A sample f 140,000 Twitter users in the battlegrund state f Michigan shared as many junk news items as prfessinal news during the final ten days f the 2016 electin. Russian gvernment interfered with the 2016 electin. Fake-news websites and bts used. Pizzagate: cnspiracy thery led t shting 4

5 Cmbating Falsehds 100+ active fact-checking sites in 2017 (PlitiFact.cm, FullFact.rg, CNN, Washingtn Pst, New Yrk Times, ) Ggle and Bing include fact-checks in search results. Facebk lets users reprt false items and flags items disputed by fact-checkers. 5

6 Fact-checking is Hard ur Navy is smaller than it's been since 1917", said Republican candidate Mitt Rmney in third presidential debate in

7 Fact-checking is Hard ur Navy is smaller than it's been since 1917", said Republican candidate Mitt Rmney in third presidential debate in vs

8 8 Demanding Task vs. Lack f Autmated Tls

9 9 The Hly Grail: Autmated, Live Fact-Checking

10 10 ClaimBuster: The First-ever End-t-end System

11 11

12 12

13 13

14 14

15 15

16 16

17 Finding Imprtant Factual Claims: A Supervised Learning Task 2016 Presidential Debates Presidential Debate Transcripts ( ) Human Anntatin Grund Truth Feature Extractin Feature Vectrs Learning Algrithm Imprtant factual claims 17

18 Dataset: Presidential Debate Transcripts Surce: All 30 debates (11 general electins) in histry: 1960, k sentences by presidential candidates: remved very shrt (< 5 wrds) sentences 18

19 Classificatin and Ranking by Check-wrthiness CFS: Imprtant factual claims We spend less n the military tday than at any time in ur histry. The President s psitin n gay marriage has changed. Mre peple are unemplyed tday than fur years ag. UFS: Unimprtant factual claims I was in Iwa yesterday. My mther enjys cking. I ran fr President nce befre. NFS: N factual claims (pinins, questins & declaratins) Iran must nt get nuclear weapns. 7% unemplyment is t high. My ppnent is wishy-washy. I will be tugh n crime. "Why shuld we d that? Hell, New Hampshire! Our plan is t reduce tax rate by 10%. 19

20 Feature Extractin Sentiment: Wrd: state, legislature, 87, percent, demcrat I was in a state where my legislature was 87 percent Demcrat. Part-f-Speech: Nun Entity Type: Quantity Cncept: United States 20 Sentiment: [-1.0 t 1.0] Wrd: tf-idf scres f 6130 wrds (excluding rare wrds) POS Tag: 43 tags Entity Type: 26 types

21 Feature Imprtance 6615 features in ttal Used a Randm Frest Classifier t calculate the imprtance f each feature. Wrd percent; peple; jbs POS Tag Entity Type Cncept nun; cardinal number; past tense; prepsitin Quantity; Cuntry; FieldTerminlgy; Persn United States Senate; Barack Obama Sentiment 21

22 Classificatin Accuracy Class Ttals Nn-factual Statement (NFS) Unimprtant Factual Statement (UFS) Cntains a Factual Statement (CFS) 1849 Algrithm Features P NFS P UFS P CFS P AVG * R NFS R UFS R CFS R AVG * F NFS F UFS F CFS F AVG * SVM Parts f Speech CNN Ggle News RNN GLVE P: Precisin R: Recall F: F1-Scre *: Average (AVG) is weighted based n the distributin f the number f labels in each class. 1: Pre-trained vectrs trained n part f Ggle News dataset (abut 100 billin wrds). The mdel cntains 300-dimensinal vectrs fr 3 millin wrds and phrases. 2: Glbal Vectrs fr Wrd Representatin using The Cmmn Crawl crpus which cntains petabytes f data cllected ver the last 7 years.

23 23 Our Own CrwdFlwer

24 24 Cder Quality

25 Grund Truth Cllectin sentences 374 cders labels 86 tp-quality cders labels Majrity vting admitted sentences sentences 20 mnths, 374 cders, ~$4,000 paid 30 training sentences 1032 screening sentences (731 NFS, 63 UFS, 238 CFS) t detect spammers & lw-quality cders Class Cunt Percentage CFS % UFS % NFS % 25

26 Examples f Challenging Cases But everybdy's affected by the debt because f the tremendus interest that ges int paying n that debt everything's mre expensive. And General Brwn has indicated uh his aplgy, his regrets, and I think that will, uh - in this situatin, settle the matter. 26

27 27 Case Study: 2016 U.S. Presidential Electin Debates

28 28 ClaimBuster: The First-ever End-t-end System

29 Onging Develpment Frnt end: Claim prtal Repsitry f claims (scial media, cngressinal recrd, news article) Taxnmy Structured Analysis Search and brwsing interface Analytics Back end: Claim understanding and algrithmic fact-checking Claim representatin and semantics Claim-t-query translatin Questin answering Claim exceptinality/rebustness 29

30 Cllabratrs 30 Current/Frmer UTA Students Fatma Arslan Jsue Caraball Naeemul Hassan Damian Jimenez Gensheng Zhang Cllabratrs Bill Adair (Duke) Pankaj Agarwal (Duke) Peter Frays (University f Technlgy Sydney) Siddhantsanjeev Gawsane Shhedul Hasan Minuml Jseph James Hamiltn (Stanfrd) Mark Tremayne (UTA) Aaditya Kulkarni Anil Kumar Nayak Vikas Sable Yu Wu (Ggle) Jun Yang (Duke) Cng Yu (Ggle)

31 Funding Spnsrs NSF IIS # Duke Tech & Check Cperative NSF I-Crps Teams # Knight Prttype Fund NSF IIS Medium #

32 32 In the Press

33 Btminatr: Autmated Detectin f Fake-news Bts Objective discver scial bts that publish and spread fake news Recently funded by the Texas Natinal Security Netwrk Interdisciplinary Apprach Scial netwrk, graph, and text mining Cde analysis News cntent and effect analysis 33