Mining the Social Web - Twitter Projects. Iza Moise, Evangelos Pournaras, Dirk Helbing

Transcription

1 Mining the Social Web - Twitter Projects Iza Moise, Evangelos Pournaras, Dirk Helbing Iza Moise, Evangelos Pournaras, Dirk Helbing 1

2 Social media 90%of today s data Fastest means of population feedback and provides cheap two-way information sharing Personal nature of the data Iza Moise, Evangelos Pournaras, Dirk Helbing 2

3 Social media 90%of today s data Fastest means of population feedback and provides cheap two-way information sharing Personal nature of the data Iza Moise, Evangelos Pournaras, Dirk Helbing 2

4 Social media 90%of today s data Fastest means of population feedback and provides cheap two-way information sharing Personal nature of the data Iza Moise, Evangelos Pournaras, Dirk Helbing 2

5 Social media 90%of today s data Fastest means of population feedback and provides cheap two-way information sharing Personal nature of the data Iza Moise, Evangelos Pournaras, Dirk Helbing 2

6 Twitter Is the 21st Century Newspaper information network more than a social network a crowdsourced news source origin: a real-time news source during the San Francisco earthquake Iza Moise, Evangelos Pournaras, Dirk Helbing 3

7 Twitter Is the 21st Century Newspaper information network more than a social network a crowdsourced news source origin: a real-time news source during the San Francisco earthquake Iza Moise, Evangelos Pournaras, Dirk Helbing 3

8 Why Twitter? Social media/microblogging service that provides a "Timeline" of user tweets Real-time, up to the minute 232 million(ish) users 460,000 new accounts per day 1 billion tweets per week (6000 tweets per second) in terms of social media and information proliferation, Twitter matters Iza Moise, Evangelos Pournaras, Dirk Helbing 4

9 Why Twitter? Social media/microblogging service that provides a "Timeline" of user tweets Real-time, up to the minute 232 million(ish) users 460,000 new accounts per day 1 billion tweets per week (6000 tweets per second) in terms of social media and information proliferation, Twitter matters Iza Moise, Evangelos Pournaras, Dirk Helbing 4

10 How Twitter works tweets 140 character statuses retweet Virality hashtags # Topics and trends Awareness followers Reach Iza Moise, Evangelos Pournaras, Dirk Helbing 5

11 Twitter as a Tool #TheDress #WhiteAndGold #BlueAndBlack #TheJacket #GreenAndGold #BlueAndWhite Iza Moise, Evangelos Pournaras, Dirk Helbing 6

12 Twitter as a Tool Arab Spring Hashtag activism Twitter may have helped Nigeria contain Ebola Iza Moise, Evangelos Pournaras, Dirk Helbing 6

13 Twitter as a Tool Arab Spring Hashtag activism Twitter may have helped Nigeria contain Ebola Iza Moise, Evangelos Pournaras, Dirk Helbing 6

14 Twitter as a Tool Arab Spring Hashtag activism Twitter may have helped Nigeria contain Ebola Iza Moise, Evangelos Pournaras, Dirk Helbing 6

15 What s in a Tweet? 140-character text BUT... as many pieces of metadata as there are characters 150 separate points of metadata in a single tweet Iza Moise, Evangelos Pournaras, Dirk Helbing 7

16 What s in a Tweet? 140-character text BUT... as many pieces of metadata as there are characters 150 separate points of metadata in a single tweet Iza Moise, Evangelos Pournaras, Dirk Helbing 7

17 {"id"=> , DEPRECATED The author's user ID. The author of the tweet. This embedded object can get out of sync. Number of tweets this user has. Whether this user has geo enabled ( The geo tag on this tweet in GeoJSON ( The tweet's unique ID. These IDs are roughly sorted & developers should treat them as opaque ( Text of the tweet. Consecutive duplicate tweets are rejected. 140 character max ( "text"=> "An early look at Annotations: Tweet's "created_at"=>"fri Apr 16 17:55: ", creation "in_reply_to_user_id"=>nil, The ID of an existing tweet that date. "in_reply_to_screen_name"=>nil, this tweet is in reply to. Won't "in_reply_to_status_id"=>nil The screen name & be set unless the author of the "favorited"=>false, referenced tweet is mentioned. "truncated"=>false, Truncated to 140 user ID of replied to "user"=> characters. Only tweet author. {"id"=> , possible from SMS. The author's user name. The author's "screen_name"=>"twitterapi", "name"=>"twitter API", The author's biography. "description"=> screen name. "The Real Twitter API. I tweet about API changes, service issues and happily answer questions about Twitter and our API. Don't get an answer? It's on my website.", "url"=>" The author's "location"=>"san Francisco, CA", URL. The author's "location". This is a free-form text field, and "profile_background_color"=>"c1dfee", there are no guarantees on whether it can be geocoded. "profile_background_image_url"=> " Rendering information "profile_background_tile"=>false, for the author. Colors "profile_image_url"=>" are encoded in hex "profile_link_color"=>"0000ff", values (RGB). "profile_sidebar_border_color"=>"87bc44", The creation date "profile_sidebar_fill_color"=>"e0ff92", for this account. "profile_text_color"=>"000000", Whether this account has "created_at"=>"wed May 23 06:01: ", contributors enabled "contributors_enabled"=>true, ( Number of "favourites_count"=>1, favorites this "statuses_count"=>1628, Number of user has. "friends_count"=>13, users this user "time_zone"=>"pacific Time (US & Canada)", The timezone and offset is following. "utc_offset"=>-28800, (in seconds) for this user. "lang"=>"en", The user's selected "protected"=>false, language. "followers_count"=>100581, "geo_enabled"=>true, Whether this user is protected "notifications"=>false, DEPRECATED or not. If the user is protected, "following"=>true, in this context Number of then this tweet is not visible "verified"=>true}, Whether this user followers for except to "friends". "contributors"=>[ ], has a verified badge. this user. "geo"=>nil, "coordinates"=>nil, DEPRECATED "place"=> The contributors' (if any) user The place ID {"id"=>"2b6ff8c22edd9576", IDs ( "url"=>" "name"=>"soma", The URL to fetch a detailed "full_name"=>"soma, San Francisco", The printable names of this place polygon for this place "place_type"=>"neighborhood", "country_code"=>"us", The type of this "country"=>"the United States of America", place - can be a The place associated with this "bounding_box"=> "neighborhood" Tweet ( {"coordinates"=> or "city" [[[ , ], [ , ], The country this place is in [ , ], [ , ]]], "type"=>"polygon"}}, "source"=>"web"} The application The bounding that sent this box for this Map of a Twitter Status Object tweet place Raffi Krikorian <raffi@twitter.com> Iza Moise, Evangelos Pournaras, Dirk Helbing 18 April

18 Twitter Data Analytics Early detection of Twitter trends Political polarisation on Twitter Controversial topics Trends Predicting flu-epidemics using Twitter Iza Moise, Evangelos Pournaras, Dirk Helbing 9

19 The Start of Web Epidemiology health-related conversations on Twitter chronic disease, disease detection, quality of patient care, gun violence etc. real-time nature of the social platform the openness of people when discussing health issues Iza Moise, Evangelos Pournaras, Dirk Helbing 10

20 Twitter and HealthMap tracking cholera after the earthquake in Haiti official reports lag two weeks Iza Moise, Evangelos Pournaras, Dirk Helbing 11

21 Further Health Examples daily city-level monitoring in order to track flu outbreaks determine drug side-effects 23 common medications (antidepressants) relevant Tweets during 6 months tweets describing side effects study atypical sleeping patterns Ebola study on Twitter Iza Moise, Evangelos Pournaras, Dirk Helbing 12

22 Geoinformation in Twitter derive travel patterns in cities racially-based spatial segregation She said Yes! changes in behaviour after major milestones Aurora Borealis spottings [page=1,width=3.5in, height=1.5in]figures/aurora.jpg Iza Moise, Evangelos Pournaras, Dirk Helbing 13

23 Geoinformation in Twitter derive travel patterns in cities racially-based spatial segregation She said Yes! changes in behaviour after major milestones Aurora Borealis spottings [page=1,width=3.5in, height=1.5in]figures/aurora.jpg Iza Moise, Evangelos Pournaras, Dirk Helbing 13

24 Geoinformation in Twitter derive travel patterns in cities racially-based spatial segregation She said Yes! changes in behaviour after major milestones Aurora Borealis spottings Iza Moise, Evangelos Pournaras, Dirk Helbing 13

25 Some Research Questions How do people self-organise using micro-blogging? (similarity of the tweets...) How do people react during crisis situations? Who are the influential users that start a topic/trend? Sentiment analysis What type of tweets are retweeted more than others? Iza Moise, Evangelos Pournaras, Dirk Helbing 14

26 How to get Twitter data? Twitter Streaming API real-time data Firehose feed 100% data providers Streaming API feed 1% yourself Query for keywords, hashtags, user ids, and geographic bounding boxes simultaneously Returns tweets matching parameters Samples data when volume reaches 1% Get the followers and friends (limited number) Iza Moise, Evangelos Pournaras, Dirk Helbing 15

27 Twitter Datasets disease related tweets (1 year) refugees tweets (2 months) unfiltered tweets Archive team Twitter dataset (30TB, ) Iza Moise, Evangelos Pournaras, Dirk Helbing 16

28 #Interested? setup a script for streaming data (define queries and obtain matching tweets) CLD C4, Clausiustrasse 37, each monday 3 5 pm izabela.moise@gess.ethz.ch Iza Moise, Evangelos Pournaras, Dirk Helbing 17