Socially-Informed Timeline Generation for Complex Events

Transcription

1 Socially-Informed Timeline Generation for Complex Events Lu Wang, Claire Cardie, and Galen Marchetti Department of Computer Science Cornell University

2 Timelines [Joseph Priestley's A New Chart of History, 1765]

3 Timelines [Timeline of Steve Jobs, by The Geeks Club]

4 Timelines Antibiotics Positive hydrogen breath test 4 months of good guts - After antibiotics for spider bite New Diet Strong Probiotics Stomach went bad - Morning issues, fine rest of day - Worse when eat carbs Double vision recurrence - Daily severe gut probs - Few recent missed nights of sleep Voice Weakness [Katie McCurdy s Medical Timeline, by Katie McCurdy]

5 Timelines [Timeline of political unrest in Ukraine, by Global Times ]

6 Crimeans vote to rejoin Russia or return to its status under the 1992 constitution. Obama declared sanctions on Russian officials. President Obama warned Vladimir Putin that further provocations could isolate Russia. March 16 th, 2014 March 17 th, 2014 March 18 th, 2014

7 Crimeans vote to rejoin Russia or return to its status under the 1992 constitution. Obama declared sanctions on Russian officials. President Obama warned Vladimir Putin that further provocations could isolate Russia. March 16 th, 2014 March 17 th, 2014 March 18 th, 2014

8 Crimeans vote to rejoin Russia or return to its status under the 1992 constitution. Obama declared sanctions on Russian officials. President Obama warned Vladimir Putin that further provocations could isolate Russia. March 16 th, 2014 March 17 th, 2014 March 18 th, 2014 Comment: Does the West and US have a policy at all? The Obama administration has warned of increasingly harsh sanctions, but it is unlikely that Europe will comply.

9 Socially-Informed Timeline Generation Crimeans vote to rejoin Russia or return to its status under the 1992 constitution. Obama declared sanctions on Russian officials. President Obama warned Vladimir Putin that further provocations could isolate Russia. March 16 th, 2014 March 17 th, 2014 March 18 th, 2014 Comment Summary Comment Summary Does the West and US have a policy at all? The Obama administration has warned of increasingly harsh sanctions, but it is unlikely that Europe will comply. Comment Summary

10 Ø Related Work Ø Methodology Outline Socially-Informed Timeline Generation Ø Experimental Setup Data Collection Ø Evaluation Intrinsic and Extrinsic Evaluation Ø Conclusion

11 Ø Related Work Ø Methodology Outline Socially-Informed Timeline Generation Ø Experimental Setup Data Collection Ø Evaluation Intrinsic and Extrinsic Evaluation Ø Conclusion

12 Related Work: Summarization with Social Context Timeline generation from news articles and comment summarization have been studied as separate tasks. [Chieu and Lee, 2004][Yan et al., 2012][Ma et al., 2012]

13 Related Work: Timeline Generation Modeling the quality of timelines Burstiness or interestingness [Chieu and Lee, 2004] Relevance, coverage, diversity, and coherence [Yan et al, 2011] Constructing the timeline Greedy algorithm [Chieu and Lee, 2004] Dynamic programming [Yan et al, 2011; Yan et al, 2012; Zhao et al, 2013]

14 Related Work: Topic detection and tracking Document-level link detection and topic tracking [Allan et al., 1998] Event threading [Nallapati et al., 2004] Coherent graph for news articles [Shahaf et al., 2012]

15 Related Work: Summarization with Social Context Article summaries informed by social context Learning users interests [Hu et al., 2008] Improving the word importance estimation [Zhao et al., 2013] Joint summarization: article + tweet Supervised learning based on factor graphs [Yang et al., 2011] Unsupervised learning with topic modeling [Gao et al., 2012]

16 Socially-Informed Timeline Generation A socially-informed timeline generation system jointly generates a news article summary and a user comment summary for each day of an ongoing complex event.

17 Challenges Importance estimation of article sentences and comments Large volumes of comments

18 Challenges Enforcing continuity in the timeline for complex event Different from document-level continuity [Shahaf et al., 2012] Crimeans vote to rejoin Russia or return to its status under the 1992 constitution. The Crimean parliament officially declared independence. Obama declared sanctions on Russian officials. President Obama warned Vladimir Putin that further provocations could isolate Russia. March 16 th, 2014 March 17 th, 2014 March 18 th, 2014

19 Challenges Connection between article summary and comment summary Obama declared sanctions on Russian officials. March 16 th, 2014 March 17 th, 2014 March 18 th, 2014 Comment Summary The Obama administration has warned of increasingly harsh sanctions, but it is unlikely that Europe will comply.

20 Challenges Importance estimation of article sentences and comments Joint learning for importance estimation Enforcing continuity in the timeline for complex event Entity-centered event threading Connection between article summary and comment summary Bipartite matching

21 Challenges Joint Learning Importance estimation of article sentences and comments Joint learning for importance estimation Enforcing continuity in the timeline for complex event Entity-centered event threading Connection between article summary and comment summary Bipartite matching

22 Challenges Timeline Generation Importance estimation of article sentences and comments Joint learning for importance estimation Enforcing continuity in the timeline for complex event Entity-centered event threading Connection between article summary and comment summary Bipartite matching

23 Ø Related Work Ø Methodology Outline Socially-Informed Timeline Generation Ø Experimental Setup Data Collection Ø Evaluation Intrinsic and Extrinsic Evaluation Ø Conclusion

24 Socially-Informed Timeline Generation How to find the salient sentences and user comments? Joint Learning for Importance Estimation How to measure the quality of a timeline with social context? Summary Quality Measurement How to construct the timeline? An Alternating Optimization framework

25 Socially-Informed Timeline Generation How to find the salient sentences and user comments? Joint Learning for Importance Estimation How to measure the quality of a timeline with social context? Summary Quality Measurement How to construct the timeline? An Alternating Optimization framework

26 Joint Learning for Importance Estimation

27 Joint Learning for Importance Estimation Training data: New York Times ,863 articles each article has a human-written abstract 833,032 comments editor s-picks or not

28 Joint Learning for Importance Estimation Training data: a set of articles Each article Sentences d i contains a set of D 1 D = {d i } i=0 Gold-standard importance score: ROUGE-2 [Lin and Hovy, 2003] Comments Gold-standard importance score: editor s picks are assigned 1.0; otherwise, 0.0

29 The Joint Learning Method Article Sentence Scoring (0.8) S 0 : The Crimean parliament officially declared independence. (0.9) S 1 : Obama declared sanctions on Russian officials. Part I Linear regression scorer Comment Scoring Part III graph-based regularization (1.0) C 0 : The Crimean Parliament, headed by an ethnic Russian separatist, has voted (1.0) C 1 : The Obama administration has warned of "increasingly harsh sanctions", but it is unlikely that Europe will comply (1.0) C 2 : Sanctions are effective and if done in unison with the EU Part II Linear regression scorer

30 The Joint Learning Method Objective Function: J(w s, w c ) = J s (w s )+ J c (w c )+ J s,c (w s, w c ) Sentence Comment

31 The Joint Learning Method Objective Function: J(w s, w c ) = J s (w s )+ J c (w c )+ J s,c (w s, w c ) Part I: Sentence importance only Loss function + Regularization Part II: Comment importance only Loss function + Regularization Part III: Interplay between sentences and comments Regularization

32 The Joint Learning Method Objective Function:!!X = # # " J(w s, w c ) = J s (w s )+ J c (w c )+ J s,c (w s, w c )!X s 0 0! Xc $ & & %!X! =!!β = β si k 0 # "# 0 β c I l " $ $ # $ & %&!X! s 0 0!X! c % ' ' &!!Y = # # "!Y s!y c "!λ = λ I s X! s 0 $ $ 0 λ c I Xc! # $ " & Y &!! = $ $ % # % ' ' & Ys!! Yc!! % ' ' &! w = # " # "!L = $ $ # $ w s w c λ sc I Xs λ sc! Q T $ & % & λ sc! Q λ sc I Xc % ' ' &' Closed Form Solution: ŵ = ( X! T L! X! + X! T!X +!X! T! λ!x! + β)! 1 ( X! T Y! +!X! T! λ!y! )

33 The Joint Learning Method Objective Function: J(w s, w c ) = J s (w s )+ J c (w c )+ J s,c (w s, w c ) Part I: Sentence importance only Loss function + Regularization Part II: Comment importance only Loss function + Regularization Part III: Interplay between sentences and comments Regularization

34 Joint Learning: Start with Ridge Regression Article Sentence Scoring (0.8) S 0 : The Crimean parliament officially declared independence. (0.9) S 1 : Obama declared sanctions on Russian officials. Part I Linear regression scorer Comment Scoring Part III graph-based regularization (1.0) C 0 : The Crimean Parliament, headed by an ethnic Russian separatist, has voted (1.0) C 1 : The Obama administration has warned of "increasingly harsh sanctions", but it is unlikely that Europe will comply (1.0) C 2 : Sanctions are effective and if done in unison with the EU Part II Linear regression scorer

35 Joint Learning: Start with Ridge Regression Article Sentence Scoring x s Sentence : feature vector Training set:!x s and Ys! Ridge regression: x s R k f s (x s ) = x s w s Objective function: min Xs! w s! 2 2 Y s + βs w s w s 2 2 Label loss L 2 regularization

36 Joint Learning: Start with Ridge Regression Article Sentence Scoring x s Sentence : feature vector Training set:! Xs Ridge regression: f s (x s ) = x s w s and Objective function:!y s min Xs! w s! 2 2 Y s + βs w s w s 2 2 Label loss L 2 regularization x s R k Basic Features Length Position Number of named entities Overlaps with the headline Average/Sum TF-IDF Social Features Average/Sum frequency of content words appearing in comments Average/Sum frequency of dependency relations appearing in comments

37 Joint Learning: Start with Ridge Regression Article Sentence Scoring (0.8) S 0 : The Crimean parliament officially declared independence. (0.9) S 1 : Obama declared sanctions on Russian officials. Part I Linear regression scorer Comment Scoring Part III graph-based regularization (1.0) C 0 : The Crimean Parliament, headed by an ethnic Russian separatist, has voted (1.0) C 1 : The Obama administration has warned of "increasingly harsh sanctions", but it is unlikely that Europe will comply (1.0) C 2 : Sanctions are effective and if done in unison with the EU Part II Linear regression scorer

38 Joint Learning: Start with Ridge Regression Comment Scoring x c Comment : feature vector Training set: Ridge regression:!x c and Yc! x c R l f c (x c ) = x c w c Objective function: min Xc! w c! 2 2 Y c + βc w c w c 2 2 Label loss L 2 regularization

39 Joint Learning: Start with Ridge Regression Basic Features Length Number of named entities Contains URL Readability Features Flesch-Kincaid Readability Gunning-Fog Readability Sentiment Features Number of positive/negative words Number of sentiment words (MPQA (Wilson et al., 2005), General Inquirer (stone et al., 1966)) Comment Scoring x c Comment : feature vector Training set: Ridge regression: f c (x c ) = x c w c!x c and Yc! Objective function: min Xc! w c! 2 2 Y c + βc w c w c 2 2 Label loss L 2 regularization x c R l

40 Joint Learning: Graph-Based Regularization Article Sentence Scoring (0.8) S 0 : The Crimean parliament officially declared independence. (0.9) S 1 : Obama declared sanctions on Russian officials. Part I Linear regression scorer Comment Scoring Part III graph-based regularization (1.0) C 0 : The Crimean Parliament, headed by an ethnic Russian separatist, has voted (1.0) C 1 : The Obama administration has warned of "increasingly harsh sanctions", but it is unlikely that Europe will comply (1.0) C 2 : Sanctions are effective and if done in unison with the EU Part II Linear regression scorer

41 Joint Learning: Graph-Based Regularization Graph-Based Regularization Adjacency (similarity) matrix N, M!A R N M are the numbers of sentences and comments d i J s,c (w s, w c ) = λ sc! Axs,x c (x s w s x c w c ) 2 x s x s di,x c x c di Similarity Sentence Comment

42 Joint Learning: Graph-Based Regularization Graph-Based Regularization Adjacency (similarity) matrix N, M are the numbers of sentences and comments d i J s,c (w s, w c ) = λ sc! Axs,x c (x s w s x c w c ) 2 x s x s di,x c x c di!a R N M (0.9) S 1 : Obama declared sanctions on Russian officials. (1.0) C 1 : The Obama administration has warned of "increasingly harsh sanctions", but it is unlikely that Europe will comply (1.0) C 2 : Sanctions are effective and if done in unison with the EU

43 Joint Learning: Graph-Based Regularization Objective Function: J(w s, w c ) = J s (w s )+ J c (w c )+ J s,c (w s, w c ) Part I: Sentence importance only Loss function + Regularization Part II: Comment importance only Loss function + Regularization Part III: Interplay between sentences and comments Regularization Specialty of first sentence

44 Joint Learning: Graph-Based Regularization Objective Function: J(w s, w c ) = J s (w s )+ J c (w c )+ J s,c (w s, w c ) Part I: Sentence importance only Loss function + Regularization Part II: Comment importance only Loss function + Regularization Part III: Interplay between sentences and comments Regularization Specialty of first sentence Bias towards editor s picks comments

45 The Joint Learning Method Objective Function:!!X = # # " J(w s, w c ) = J s (w s )+ J c (w c )+ J s,c (w s, w c )!X s 0 0! Xc $ & & %!X! =!!β = β si k 0 # "# 0 β c I l " $ $ # $ & %&!X! s 0 0!X! c % ' ' &!!Y = # # "!Y s!y c "!λ = λ I s X! s 0 $ $ 0 λ c I Xc! # $ " & Y &!! = $ $ % # % ' ' & Ys!! Yc!! % ' ' &! w = # " # "!L = $ $ # $ w s w c λ sc I Xs λ sc! Q T $ & % & λ sc! Q λ sc I Xc % ' ' &' Closed Form Solution: ŵ = ( X! T L! X! + X! T!X +!X! T! λ!x! + β)! 1 ( X! T Y! +!X! T! λ!y! )

46 Socially-Informed Timeline Generation A socially-informed timeline generation system jointly generates a news article summary and a user comment summary for each day of an ongoing complex event. For each day Input: a set of articles with sentence set and comment set V c Output: article summary comment summary C V c S V s and V s

47 Socially-Informed Timeline Generation How to find the salient sentences and user comments? Joint Learning for Importance Estimation How to measure the quality of a timeline with social context? Summary Quality Measurement How to construct the timeline? An Alternating Optimization framework

48 Socially-Informed Timeline Generation 1. Importance 2. Coverage 3. Continuity The Crimean parliament officially declared independence. Crimeans vote to Obama declared sanctions on rejoin Russia Russian officials President Obama warned Vladimir Putin March 16 th, 2014 March 17 th, 2014 March 18 th, 2014 Comment Summary Comment Summary The Obama administration has warned of increasingly harsh sanctions, but it is unlikely that Europe will comply. Comment Summary

49 Socially-Informed Timeline Generation Crimeans vote to rejoin Russia The Crimean parliament officially declared independence. Obama declared sanctions on Russian officials President Obama warned Vladimir Putin March 16 th, 2014 March 17 th, 2014 March 18 th, 2014 Comment Summary Comment Summary The Obama administration has warned of increasingly harsh sanctions, but it is unlikely that Europe will comply. Importance/Insightfulness Comment Summary

50 Socially-Informed Timeline Generation Crimeans vote to rejoin Russia Connectivity The Crimean parliament officially declared independence. Obama declared sanctions on Russian officials President Obama warned Vladimir Putin March 16 th, 2014 March 17 th, 2014 March 18 th, 2014 Comment Summary Comment Summary The Obama administration has warned of increasingly harsh sanctions, but it is unlikely that Europe will comply. Comment Summary

51 Socially-Informed Timeline Generation An optimization framework Z(S,C;T ) = S qual (S;T )+C qual (C)+δX (S,C) S C T : Article Summary : Comment Summary : Event Threads

52 Socially-Informed Timeline Generation An optimization framework Z(S,C;T ) = S qual (S;T )+C qual (C)+δX (S,C) S C T : Article Summary : Comment Summary : Event Threads Constraints: length(s) θ s size(c) θ c

53 Summary Quality Measurement Z(S,C;T ) = S qual (S;T )+C qual (C)+δX (S,C) Article Summary Importance Coverage Comment Summary Continuity Importance Connectivity

54 Summary Quality Measurement: Importance Z(S,C;T ) = S qual (S;T )+C qual (C)+δX (S,C) Article Summary Importance Coverage Comment Summary Continuity Importance Connectivity

55 Summary Quality Measurement: Importance Z(S,C;T ) = S qual (S;T )+C qual (C)+δX (S,C) Article Summary Importance Coverage Comment Summary Continuity Importance Connectivity simply add up the importance scores

56 Summary Quality Measurement: Coverage Z(S,C;T ) = S qual (S;T )+C qual (C)+δX (S,C) Article Summary Importance Coverage Comment Summary Continuity Importance Connectivity

57 Summary Quality Measurement: Coverage All Candidate Sentences S 0 : The Crimean declared independence. S 1 : Obama declared sanctions. S 2 : EU condemned Russia. Current Summary The Crimean declared independence. Obama declared sanctions. S 3 : Russia recognized the independence.

58 Summary Quality Measurement: All Candidate Sentences Coverage S 0 : The Crimean declared independence. S 1 : Obama declared sanctions. S 2 : EU condemned Russia. S 3 : Russia recognized the independence. s! V s s S tfidf (s, s!) Current Summary The Crimean declared independence. Obama declared sanctions. Each sentence s Coverage of s by current summary S

59 Summary Quality Measurement: All Candidate Sentences Coverage S 0 : The Crimean declared independence. S 1 : Obama declared sanctions. S 2 : EU condemned Russia. S 3 : Russia recognized the independence. Current Summary The Crimean declared independence. Obama declared sanctions. s! V s s S ŝ V s min( tfidf (s, s!),α tfidf (ŝ, s!)) Each sentence s Coverage of s by current summary S Upper bound for encourage diversity [Lin and Bilmes, 2011]

60 Summary Quality Measurement: Continuity Z(S,C;T ) = S qual (S;T )+C qual (C)+δX (S,C) Article Summary Importance Coverage Comment Summary Continuity Importance Connectivity

61 Summary Quality Measurement: Continuity An optimization framework Event Threads Z(S,C;T ) = S qual (S;T )+C qual (C)+δX (S,C) Article Summary Importance Coverage Comment Summary Continuity Importance Connectivity Entity-Centered Event Threading

62 Day 1: 1. Barack Obama pledges to stand with Ukraine. 2. Putin formally asked the federation council for permission to use the armed forces in Ukraine.

63 Day 1: 1. Barack Obama pledges to stand with Ukraine. 2. Putin formally asked the federation council for permission to use the armed forces in Ukraine. τ 1 * Barack Obama pledges to τ 2 * Putin formally asked the

64 Day 2: 1. Obama declared sanctions on Russian officials. 2. US president Barack Obama urges Moscow to move back its troops and lower tensions. 3. Putin addresses parliament, defending Moscow s actions on Crimea. τ 1 * Barack Obama pledges to τ 2 * Putin formally asked the

65 Day 2: 1. Obama declared sanctions on Russian officials. 2. US president Barack Obama urges Moscow to move back its troops and lower tensions. 3. Putin addresses parliament, defending Moscow s actions on Crimea. τ 1 * Barack Obama pledges to * Obama declared sanctions on * US President urges Moscow τ 2 * Putin formally asked the * Putin addresses parliament

66 Day 3: 1. Russian president Vladimir Putin dismisses claims that Russian agents are acting in eastern Ukraine. 2. Barack Obama extends the list of individuals targeted for sanctions. 3. EU rejected the referendum on independence in Crimea. τ 1 * Barack Obama pledges to * Obama declared sanctions on * US President urges Moscow τ 2 * Putin formally asked the * Putin addresses parliament

67 Day 3: 1. Russian president Vladimir Putin dismisses claims that Russian agents are acting in eastern Ukraine. 2. Barack Obama extends the list of individuals targeted for sanctions. 3. EU rejected the referendum on independence in Crimea. τ 1 * Barack Obama pledges to * Obama declared sanctions on * US President urges Moscow * Barack Obama extends the list τ 2 * Putin formally asked the * Putin addresses parliament * Russian president Vladimir Putin

68 Day 3: 1. Russian president Vladimir Putin dismisses claims that Russian agents are acting in eastern Ukraine. 2. Barack Obama extends the list of individuals targeted for sanctions. 3. EU rejected the referendum on independence in Crimea. τ 1 * Barack Obama pledges to * Obama declared sanctions on * US President urges Moscow * Barack Obama extends the list τ 2 * Putin formally asked the * Putin addresses parliament * Russian president Vladimir Putin τ 3 * EU rejected the referendum on

69 Summary Quality Measurement: τ 1 Continuity Existing Event Threads * Barack Obama pledges to * Obama declared sanctions on * US President urges Moscow Current Summary * Barack Obama extends the list τ 2 * Putin formally asked the * Putin addresses parliament * Russian president Vladimir Putin τ 3 * EU rejected the referendum on τ T Each thread max s k S cohesion(s k,τ ) Continuity for τ: maximum cohesion of any sentence in current summary to thread, cohesion(s,τ ) = min s i τ,s i tfsimi(s i, s)

70 Summary Quality Measurement: τ 1 Continuity Existing Event Threads * Barack Obama pledges to * Obama declared sanctions on * US President urges Moscow Current Summary * Barack Obama extends the list τ 2 * Putin formally asked the * Putin addresses parliament * Russian president Vladimir Putin τ 3 * EU rejected the referendum on τ T Each thread max s k S cohesion(s k,τ ) Continuity for τ: maximum cohesion of any sentence in current summary to thread, cohesion(s,τ ) = min s i τ,s i tfsimi(s i, s)

71 Summary Quality Measurement: Connectivity Z(S,C;T ) = S qual (S;T )+C qual (C)+δX (S,C) Article Summary Importance Coverage Comment Summary Continuity Importance Connectivity

72 Summary Quality Measurement: Connectivity Goals: Comments should be on the topics in article summary. Comments that cover diversified events are preferred.

73 Summary Quality Measurement: Connectivity Article Summary S S 0 : The Crimean parliament officially declared independence Comment Summary C C 0 : The Crimean Parliament, headed by an ethnic Russian separatist, has voted 0.3 S 1 : Obama declared sanctions on Russian officials Step 1: construct a bipartite graph C 1 : The Obama administration has warned of "increasingly harsh sanctions C 2 : Sanctions are effective and if done in unison with the EU

74 Summary Quality Measurement: Connectivity Article Summary S S 0 : The Crimean parliament officially declared independence Comment Summary C C 0 : The Crimean Parliament, headed by an ethnic Russian separatist, has voted 0.3 S 1 : Obama declared sanctions on Russian officials Step 1: construct a bipartite graph Step 2: find the best matching C 1 : The Obama administration has warned of "increasingly harsh sanctions C 2 : Sanctions are effective and if done in unison with the EU

75 Summary Quality Measurement: Connectivity Article Summary S S 0 : The Crimean parliament officially declared independence Comment Summary C C 0 : The Crimean Parliament, headed by an ethnic Russian separatist, has voted 0.3 S 1 : Obama declared sanctions on Russian officials Step 1: construct a bipartite graph Step 2: find the best matching Step 3: compute the connectivity C 1 : The Obama administration has warned of "increasingly harsh sanctions C 2 : Sanctions are effective and if done in unison with the EU

76 Socially-Informed Timeline Generation Z(S,C;T ) = S qual (S;T )+C qual (C)+δX (S,C) Article Summary Importance Coverage Comment Summary Continuity Importance Connectivity

77 Socially-Informed Timeline Generation How to find the salient sentences and user comments? Joint Learning for Importance Estimation How to measure the quality of a timeline with social context? Summary Quality Measurement How to construct the timeline? An Alternating Optimization framework

78 Socially-Informed Timeline Generation An Alternating Optimization Algorithm Z(S,C;T ) = S qual (S;T )+C qual (C)+δX (S,C) Idea: alternatively search for better S and until convergence For each iteration C S S C Fix, find with hill-climbing search Fix, find with Ford-Fulkerson algorithm [Ford-fulkerson, 1956] max-weight bipartite graph matching problem à max network flow problem [Kleinberg and Tardos, 2005] C

79 Alternating Optimization Process Initialization - S 0 by greedy algorithm - C 0 by Ford-Fulkerson algorithm Article Summary S1: The Crimean parliament declared independence. S2: Barack Obama pledges to stand with Ukraine. Comment Summary C1: The Crimean Parliament, headed by an ethnic Russian separatist, voted for Crimea to be annexed into Russia... C2: Obama need to stick with America's domestic problems

80 Alternating Optimization Process Initialization - S 0 by greedy algorithm - C 0 by Ford-Fulkerson algorithm Iterations - S t by Hill-climbing search Article Summary S1: The Crimean parliament declared independence. S2: Barack Obama pledges to stand with Ukraine. S2: EU rejected the referendum on independence in Crimea. Comment Summary C1: The Crimean Parliament, headed by an ethnic Russian separatist, voted for Crimea to be annexed into Russia... C2: Obama need to stick with America's domestic problems

81 Alternating Optimization Process Initialization - S 0 by greedy algorithm - C 0 by Ford-Fulkerson algorithm Article Summary S1: The Crimean parliament declared independence. S2: EU rejected the referendum on independence in Crimea. Iterations - S t by Hill-climbing search Comment Summary C1: The Crimean Parliament, headed by an ethnic Russian separatist, voted for Crimea to be annexed into Russia... C2: Obama need to stick with America's domestic problems

82 Alternating Optimization Process Initialization - S 0 by greedy algorithm - C 0 by Ford-Fulkerson algorithm Iterations - S t by Hill-climbing search - C t by Ford-Fulkerson algorithm Article Summary S1: The Crimean parliament declared independence. S2: EU rejected the referendum on independence in Crimea. Comment Summary C1: The Crimean Parliament, headed by an ethnic Russian separatist, voted for Crimea to be annexed into Russia... C2: Obama need to stick with America's domestic problems... C2: The Obama administration warned of harsh sanctions", but it is unlikely that EU will comply...

83 Alternating Optimization Process Initialization - S 0 by greedy algorithm - C 0 by Ford-Fulkerson algorithm Article Summary S1: The Crimean parliament declared independence. S2: EU rejected the referendum on independence in Crimea. Iterations - S t by Hill-climbing search - C t by Ford-Fulkerson algorithm Comment Summary C1: The Crimean Parliament, headed by an ethnic Russian separatist, voted for Crimea to be annexed into Russia... C2: The Obama administration warned of harsh sanctions", but it is unlikely that EU will comply...

84 Ø Related Work Ø Methodology Outline Socially-Informed Timeline Generation Ø Experimental Setup Data Collection Ø Evaluation Intrinsic and Extrinsic Evaluation Ø Conclusion

85 Data Articles and User Comments Sources New York Times (NYT) CNN BBC Events Missing Malaysia Airlines flight MH370 Political unrest in Ukraine Israel-Gaza conflict NSA surveillance leaks

86 Data Time Span # Articles # Comments Missing Flight MH370 03/08-06/ ,646 Ukraine Crisis 03/08-06/30 3, ,961 Israel-Gaza Conflict 07/20-09/ ,244 NSA surveillance leaks 03/23-06/ ,481

87 Ø Related Work Ø Methodology Outline Socially-Informed Timeline Generation Ø Experimental Setup Data Collection Ø Evaluation Intrinsic and Extrinsic Evaluation Ø Conclusion

88 Experiments Question 1: Article Summary Quality Do socially-informed timeline generation systems produce more informative summaries? Question 2: Comment Summary Quality Do comment summaries provide additional insights? Question 3: Event Thread Quality Can event threads allow users to easily locate and absorb information?

89 Experiment 1: Article Summary Quality Gold-standard timelines Wikipedia pages New York Times/BBC topic page Automatic evaluation metric ROUGE scores [Lin and Hovy, 2003] : ngrams recall Comparisons Chieu and Lee, 2004: burstiness or interestingness Yan et al, 2011: coverage, diversity, and coherence Single-article abstracts

90 Experiment 1: Article Summary Quality Missing Flight Comparisons: no social context Ukraine Crisis Israel -Gaza NSA Chieu and Lee Yan et al Single-article abstract Our Systems No Alternating OPT With Alternating OPT ROUGE scores ( 100): larger numbers à better performance

91 Experiment 2: Comment Summary Quality Amazon Mechanical Turk on ranking tasks Informativeness Insightfulness Comparisons: Random Users picks (ranked by positive ratings) Editors picks 15 randomly selected timelines Each is evaluated by four people

92 Experiment 2: Comment Summary Quality Informativeness Insightfulness % Best Average Rank % Best Average Rank Random 1.7% % 3.58 User s-picks 5.0% % 2.55 Editor s picks 26.7% % 2.22 Our system 66.7% % 1.65 Inter-annotator agreement by Krippendorff's α Informativeness: 0.63 Insightfulness: 0.50

93 Experiment 3: Event Thread Quality 10-day timeline is randomly selected per dataset. Design one question describe the activities for searching for the missing flight MH undergraduate and graduate students (native speakers of English): Read the question and the timeline for 5 minutes Write down an answer after removing the timeline For each student, 2 timelines are displayed with threads, and the other 2 are not.

94 Experiment 3: Event Thread Quality Amazon Mechanical Turk to evaluate the informativeness of students answers read all 10 answers for the same question Rate informativeness on 1-to-5 scale Answer Type Mean ± STD Rated 5 (%) Rated 4 (%) No Thread 2.6 ± 1.2 7% 23% With Thread 3.3 ± % 26%

95 Article Summary March 17 th, 2014 Obama administration froze the U.S. assets of seven Russian officials, while similar sanctions were imposed on four Ukrainian officials March 18 th, 2014 Ukraine does not recognize a treaty signed in Moscow on Tuesday making its Crimean peninsula a part of Russia March 19 th, 2014 The head of NATO warned on Wednesday that Russian President Vladimir Putin may not stop with the annexation of Crimea March 20 th, 2014 The United States on Thursday expanded its sanctions on Russians Comment Summary Theodore Roosevelt said that the worst possible thing you can do in diplomacy is soft hitting. That is what the US and the EU are doing in these timid sanctions Though there were many in Crimea who supported annexation, there were certainly some who did not. What about those people?... If you look at a real map, Crimea is an island and has always been more connected to Russia than to Ukraine The US and EU should follow up economic sanctions with concrete steps to strengthen NATO

96 Conclusion We present a socially-informed timeline generation system. We describe a joint learning-based approach to identify important sentences and comments. We propose an alternating optimization framework to construct topic-relevant timelines.

97 Thank you! Questions? Datasets: