NLP Approaches to Fact Checking and Fake News Detection

Size: px

Start display at page:

Download "NLP Approaches to Fact Checking and Fake News Detection"

Abel Warner
5 years ago
Views:

1 NLP Approaches to Fact Checking and Fake News Detection Andreas Hanselowski, Iryna Gurevych

2 Outline: 1. Fake News Detection 2. Automated Fact Checking 2

3 Outline: 1. Fake News Detection 2. Automated Fact Checking 3

Fake News Detection: Fake News Challenge General idea: The Fake News Challenge is a competition in the spirit of the Netflix Prize designed to foster the development

4 Fake News Detection: Fake News Challenge General idea: The Fake News Challenge is a competition in the spirit of the Netflix Prize designed to foster the development of AI technology to help solve the fake news problem. Dean Pomerleau A number of competitions to explore AI approaches for different problems in fake news detection 4

Dean Pomerleau A number of competitions to explore AI approaches for different problems in fake news detection Fake News Challenge Stage

5 Fake News Detection: Fake News Challenge General idea: The Fake News Challenge is a competition in the spirit of the Netflix Prize designed to foster the development of AI technology to help solve the fake news problem. Dean Pomerleau A number of competitions to explore AI approaches for different problems in fake news detection Fake News Challenge Stage One: Stance classification Given: An article headline and an article body Headline To be determined: Stance of the article with respect to the headline Classes: agrees, disagrees, discusses, unrelated 5

6 Fake News Detection: Fake News Challenge General idea: The Fake News Challenge is a competition in the spirit of the Netflix Prize designed to foster the development of AI technology to help solve the fake news problem. Dean Pomerleau A number of competitions to explore AI approaches for different problems in fake news detection Fake News Challenge Stage One: Stance classification Given: An article headline and an article body Headline To be determined: Stance of the article with respect to the headline Classes: agrees, disagrees, discusses, unrelated 6

97 UCL Machine Reading 81.72 Chips Ahoy! 80.21 CLUlings 79.

7 Fake News Detection: Fake News Challenge Results: Top 10 out of 50 Team % FNC-1 score SOLAT in the SWEN Athene UKP UCL Machine Reading Chips Ahoy! CLUlings Unconscious bias OSU MITBusters DFKI LT GTRI - ICL

Fake-News Detection: Fake News Challenge Our takeaway: Data set relatively small only 1500 individual articles Poor generalization to new corpora Approaches which work well: Topic models, bag of

8 Fake-News Detection: Fake News Challenge Our takeaway: Data set relatively small only 1500 individual articles Poor generalization to new corpora Approaches which work well: Topic models, bag of words features, n-grams, tf-idf Approaches which do not work: Features based on word embeddings, RNNs (LSTMs) Problem setting is relatively simple, however for better performance larger corpora required Graduiertenkolleg Adaptive Informationsaufbereitung aus heterogenen Quellen Andreas Hanselowski

9 Outline 1. Fake News Detection 2. Automated Fact Checking 9

10 Automated Fact Checking (Claim Validation) Claim: Israel caused flooding in Gaza by opening river dams Evidence documents: The Daily Mail: Hundreds of Palestinians left homeless after Israel opens river dams and floods houses Al Jazeera: "Israel opened water dams, without warning, last night, causing serious damage to Gazan villages near the border, " General Al-Saudi told Al Jazeera. The Jerusalem post: The Daily Mail published a story on Monday that originally accused Israel of intentionally opening dams. in order to flood Gaza. The only problem is, there are no dams in southern Israel. after amending the article's headline the pretense for the charge remained in the Daily Mail article Claim is false 10

11 Automated Fact Checking (Claim Validation): NLP-Pipeline NLP Pipeline Document retrieval Claim Large corpus, Web Extraction of evidence Evidence 1 Evidence 2 Claim validation Evidence 1 Evidence 2 Verdict Claim 11

12 Automated Fact Checking (Claim Validation): Training-Data Israel caused flooding in Gaza by opening river dams Claim and Verdict Evidence text snippets Articles containing evidence Israel caused flooding in Gaza In the wake Israel of a reportedly recent severe flooded winter Gaza storm in the region. Israel reportedly flooded Gaza Israeli authorities No casualties opened were the reported floodgates as a to discharge result, the but accumulated more than 80 water families had No casualties were reported as a flooding villages to flee after in Gaza. their homes filled with result, but more than 80 families had No casualties water were levels reported sometimes as a reaching result, to flee after their homes filled with but more more. than than 80 families three meters, had to flee the Gaza water levels sometimes reaching after. their Ministry homes of filled Interior with said water in levels a more than three meters, the Gaza sometimes statement. reaching The more flood than caused three by Ministry of Interior said in a meters, the Israel Gaza forced Ministry the closure of Interior of the said main statement. The flood caused by in a statement. road connecting al-mughraqa district Israel forced the closure of the main The flood and caused Nusseirat by Israel refugee forced camp the south of road connecting al-mughraqa district closure of Gaza the main city. road connecting aland Nusseirat refugee camp south of Mughraqa In district the wake and of Nusseirat a recent severe refugee winter Gaza city. camp south storm of in Gaza the city. region, Israeli authorities In the wake of a recent severe winter opened the floodgates to discharge. storm in the region, Israeli authorities the accumulated water flooding opened the floodgates to discharge villages in Gaza. the accumulated water flooding villages in Gaza.

Automated Fact Checking (Claim Validation): Training-Data Israel caused flooding in Gaza by opening river dams Articles containing evidence Israel caused flooding in Gaza Claim and Verdict Evidence

Israel reportedly flooded Gaza Israeli authorities No casualties opened were the reported floodgates as a to discharge result, the but accumulated more than 80 water families had No casualties were

13 Automated Fact Checking (Claim Validation): Training-Data Israel caused flooding in Gaza by opening river dams Articles containing evidence Israel caused flooding in Gaza Claim and Verdict Evidence text snippets Corpus statistics: Number of claims: 3800 Number of documents: Corpus will be released f In the wake Israel of a reportedly recent severe flooded winter Gaza storm in the region. Israel reportedly flooded Gaza Israeli authorities No casualties opened were the reported floodgates as a to discharge result, the but accumulated more than 80 water families had No casualties were reported as a flooding villages to flee after in Gaza. their homes filled with result, but more than 80 families had No casualties water were levels reported sometimes as a reaching result, to flee after their homes filled with but more more. than than 80 families three meters, had to flee the Gaza water levels sometimes reaching after. their Ministry homes of filled Interior with said water in levels a more than three meters, the Gaza sometimes statement. reaching The more flood than caused three by Ministry of Interior said in a meters, the Israel Gaza forced Ministry the closure of Interior of the said main statement. The flood caused by in a statement. road connecting al-mughraqa district Israel forced the closure of the main The flood and caused Nusseirat by Israel refugee forced camp the south of road connecting al-mughraqa district closure of Gaza the main city. road connecting aland Nusseirat refugee camp south of Mughraqa In district the wake and of Nusseirat a recent severe refugee winter Gaza city. camp south storm of in Gaza the city. region, Israeli authorities In the wake of a recent severe winter opened the floodgates to discharge. storm in the region, Israeli authorities the accumulated water flooding opened the floodgates to discharge villages in Gaza. the accumulated water flooding villages in Gaza.

14 First Results (work-in-progress) Evidence extraction: Random baseline: F1= 10% Deep Learning method (LSTMs): F1 = 20% Further Deep Learning structures need to be explored Upper bound low due to annotation by one single fact checker 14

checker Claim validation: Majority baseline: F1 = 45% Deep Learning method (LSTMs): F1 = 63% Further DL structures

15 First Results (work-in-progress) Evidence extraction: Random baseline: F1= 10% Deep Learning method (LSTMs): F1 = 20% Further Deep Learning structures need to be explored Upper bound low due to annotation by one single fact checker Claim validation: Majority baseline: F1 = 45% Deep Learning method (LSTMs): F1 = 63% Further DL structures to be explored Reasoning, world knowledge needs to be incorporated Israel caused flooding in Gaza by opening river dams 15

16 Conclusions: Stance classification for fake news detection works relatively well, fact checking is more challenging Problems: Not enough data Snopes corpus is a staring point For comprehensive claim validation, reasoning, world knowledge is required Our approach: Brake down the problem into several sub-problems and tackle them individually In addition to state-of-the-art deep learning approaches, incorporation of world knowledge and reasoning techniques 16

17 DATUM Graduiertenkolleg Adaptive Informationsaufbereitung aus heterogenen Quellen 17

18 Automated Fact Checking (Claim Validation) Claim: Israel caused flooding in Gaza by opening river dams Evidence documents: The Daily Mail: Hundreds of Palestinians left homeless after Israel opens river dams and floods houses Al Jazeera: "Israel opened water dams, without warning, last night, causing serious damage to Gazan villages near the border, " General Al-Saudi told Al Jazeera. 18

Automated Fact Checking (Claim Validation) Claim: Israel caused flooding in Gaza by opening river dams Evidence documents: The Daily Mail: Hundreds of Palestinians left homeless after Israel opens

19 Automated Fact Checking (Claim Validation) Claim: Israel caused flooding in Gaza by opening river dams Evidence documents: The Daily Mail: Hundreds of Palestinians left homeless after Israel opens river dams and floods houses Al Jazeera: "Israel opened water dams, without warning, last night, causing serious damage to Gazan villages near the border, " General Al-Saudi told Al Jazeera. The Jerusalem post: The Daily Mail published a story on Monday that originally accused Israel of intentionally opening dams. in order to flood Gaza. The only problem is, there are no dams in southern Israel. after amending the article's headline the pretense for the charge remained in the Daily Mail article 19

20 Challenging Cases: Entity matching: Claim: A restaurant in South Africa was shut down for serving human meat. Evidence: A Nigerian restaurant has been shuttered by authorities for serving human flesh. Inference: Claim: Fritz Jahr introduced the concept of bioethics. Evidence: E1: The concept of bioethics was introduced in the article Wissenschaft von Sittenlehre. E2: Fritz Jahr has authored Wissenschaft von Sittenlehre. DATUM Graduiertenkolleg Adaptive Informationsaufbereitung aus heterogenen Quellen 20

21 Fake-News Detection: Fake News Challenge Model structure Model: Multi Layer Perceptron agrees disagrees discusses unrelated Features: Basic features Bag-of-words N-grams PDIG2 Topic models Latent semantic indexing (LSA) Latent Dirichlet allocation (LDA) Lexica Refuting words (fake, hoax, ) Headline feature vector Joint feature vector Docu. feature vector 21

22 Folie 21 PDIG2 was ist bag-of-words unigrams? wie unterscheiden sich diese von "normalen" unigrams? Prof. Dr. Iryna Gurevych;

Automated Fact Checking (Claim Validation): Extraction of Evidence using LSTMs Not an Evidence Evidence LSTM-2 Sentence representations: Claim

23 Automated Fact Checking (Claim Validation): Extraction of Evidence using LSTMs Not an Evidence Evidence LSTM-2 Sentence representations: Claim representation LSTM-1 Word embeddings: Al-Saudi reports in an interview. Israeli authorities opened river damps. Israel caused flooding in Gaza 22

24 Automated Fact Checking (Claim Validation): Claim Validation using LSTMs Verdict: False/True Multi Layer Perceptron LSTM Claim Evidence 1 Source 1 Evidence n Source n Israeli authorities opened river damps to flood Gaza. There are no dams in southern Israel. 23

Overview. Ø Neural Networks are considered black-box models Ø They are complex and do not provide much insight into variable relationships

Overview. Ø Neural Networks are considered black-box models Ø They are complex and do not provide much insight into variable relationships Neural Networks Overview Ø s are considered black-box models Ø They are complex and do not provide much insight into variable relationships Ø They have the potential to model very complicated patterns