Natural Language Technologies for E-Rulemaking. Claire Cardie Department of Computer Science Cornell University

Transcription

1 Natural Language Technologies for E-Rulemaking Claire Cardie Department of Computer Science Cornell University

2 An E-Rulemaking Scenario Summarize the public commentary regarding the prohibition of potassium hydroxide for peeling peaches" , letters, editorials, technical reports, newswires multi-document summary speech understanding

3 An E-Rulemaking Scenario Summarize the public commentary regarding the prohibition of potassium hydroxide for peeling peaches" , letters, editorials, technical reports, newswires multi-document summary ad hoc IR

4 An E-Rulemaking Scenario Summarize the public commentary regarding the prohibition of potassium hydroxide for peeling peaches" multi-lingual , letters, editorials, technical reports, newswires multi-document summary machine translation cross-lingual IR

5 An E-Rulemaking Scenario Summarize the public commentary regarding the prohibition of potassium hydroxide for peeling peaches" multi-lingual , letters, editorials, technical reports, newswires multi-document summary document clustering

6 An E-Rulemaking Scenario Summarize the public commentary regarding the prohibition of potassium hydroxide for peeling peaches" multi-lingual , letters, editorials, technical reports, newswires con pro pro multi-document summary text categorization

7 An E-Rulemaking Scenario Summarize the public commentary regarding the prohibition of potassium hydroxide for peeling peaches" multi-lingual , letters, editorials, technical reports, newswires con pro pro multi-document summary information extraction commenter: J. Dougherty organization: Stonyfield Farms regulation:

8 An E-Rulemaking Scenario Summarize the public commentary regarding the prohibition of potassium hydroxide for peeling peaches" multi-lingual , letters, editorials, technical reports, newswires con pro pro multi-document summary multi-document summarization commenter: J. Dougherty organization: Stonyfield Farms regulation:

9 An E-Rulemaking Scenario Summarize the public commentary regarding the prohibition of potassium hydroxide for peeling peaches" multi-lingual , letters, editorials, technical reports, newswires con pro pro multi-document summary question answering commenter: J. Dougherty organization: Stonyfield Farms regulation:

10 Ad-hoc Information Retrieval Topic: Advantages and disadvantages of using potassium hydroxide in any aspect of organic farming, especially doc 1 score doc 2 score doc 3 score doc n score information need text collection relevant documents (ranked) IR system

11 Ad-hoc IR.state of the art TReC-8 evaluation (1999) recall measure of coverage precision measure of accuracy

12 Text Categorization E.D. And F. MAN TO BUY INTO HONG KONG FIRM The U.K. Based commodity house E.D. And F. Man Ltd and Singapore s Yeo Hiap Seng Ltd jointly announced that Man will buy a substantial stake in Yeo s 71.1 pct held unit, Yeo Hiap Seng Enterprises Ltd. Man will develop the locally listed soft drinks Is the document about plants? sports? health and fitness? corporate acquisitions? stock market? No No No YES No

13 Text Categorization...state of the art accuracy ~87% ~90% ~72% Reuters Collection Reuters newswire articles 90 categories ~3300 test documents WebKB Collection WWW pages classified by function» e.g. personal home page, project home page 4 categories ~225 test documents Ohsumed MeSH Medical abstracts classified by subject heading 20 categories from disease subtree ~10,000 test documents

14 Information Extraction text collection Information Extraction System Who: What: Who: Where: What: Who: When: Where: What: How: When: Where: How: When: How:

15 IE System for Natural Disasters Disaster Type: earthquake location: Afghanistan date: 05/30/1998 magnitude: 6.9 epicenter: a remote part of the country damage: human-effect: victim: Thousands of people number: Thousands outcome: dead physical-effect: object: entire villages outcome: damaged PAKISTAN MAY BE PREPARING FOR ANOTHER TEST Thousands of people are feared dead following... (voice-over)...a powerful earthquake that hit Afghanistan today. The quake registered 6.9 on the Richter scale, centered in a remote part of the country. (on camera) Details now hard to come by, but reports say entire villages were buried by the quake. Document no.: ABC Date/time: 05/30/ :35:42.49

16 Information Extraction..state of the art Performance varies widely depending on the domain and on the style of the texts Complex events, unrestricted text (e.g. information on natural disasters from newswires):» 40-60% recall» 50-70% precision Entity extraction, semi-structured text (e.g. extraction from job ads on company web pages)» ~80-90% recall and precision

17 Information Extraction...state of the art Named Entity identification named locations, named persons, named organizations, dates, times, monetary amounts, and percentages Good performance: 90-93% recall and precision

18 Question Answering? answer + supporting text text collection What was the name of the enchanter in the movie Monty Python and the Holy Grail? Tim

19 Sample Questions How many calories are there in a Big Mac? Who is the voice of Miss Piggy? Who was the first American in space? What two US biochemists won the Nobel Prize in medicine in 1992? How many Grand Slam titles did Bjorn Borg win? Where is the Taj Mahal? What costume designer decided that Michael Jackson should only wear one glove?

20 Question Answering..state of the art Simplifications short-answer, fact-based questions answer exists in the collection as a text fragment supporting info can be found in a single document system returns up to 5 guesses per question Trec-8 (1999) Trec-9 (2000) Trec-10 (2001) # of questions collection size 528,000 (2Gb) 979,000 (3Gb) 979,000 (3Gb) % correct 70% 65% 70% avg answer rank

21 Summarization Produce a summary of a document under specific length constraints, e.g. 100 words, 10% of document length Different kinds of summaries» generic» query-dependent summary» extracts vs. generated text Evaluation is difficult. Typical metrics measure:» coverage vs. manually generated reference summaries» conciseness» grammaticality» coherence» organization

22 Multi-Document Summarization [White et al., 2002]

23 Multi-Document Summarization Biographical summary [Lin and Hovy, DUC 2002]

24 Opinion-oriented Extraction, Summarization, and QA The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. [Wiebe et al., 2003]

25 Opinion-oriented Extraction, Summarization, and QA <writer>: ATTITUDE value: neg strength: medium <HR report> <many countries>: ATTITUDE value: neg strength: high <Chinese HR report>: ATTITUDE value: neg strength: medium <USA>

26 Summary Range and current capabilities of Natural Language Technologies Ad-hoc information retrieval Text categorization Information extraction Summarization Question answering Near-term research and development possibilities Processing opinion-oriented texts