Use of Automated Writing Evaluation (AWE) for placement tests: Can scores of AWE be criteria to place students into language courses?

Size: px

Start display at page:

Download "Use of Automated Writing Evaluation (AWE) for placement tests: Can scores of AWE be criteria to place students into language courses?"

Randolph Garrett
5 years ago
Views:

1 Use of Automated Writing Evaluation (AWE) for placement tests: Can scores of AWE be criteria to place students into language courses? Zhi Li, Hyejin Yang, Stephanie Link, Volker Hegelheimer IOWA STATE UNIVERSITY October 5-6, 2012 University of Illinois, Urbana-Champaign 1

2 English Placement Test Ø To place international students into appropriate ESL writing classes Ø Practical needs Low cost options Immediate scoring Improved time management

3 Example: Time Management Fall ISU: 500+ essays to score 1 human rater: 5 min/essay How long will it take for raters to score all the essays? raters: 28 essays = 2.3 hours Each essay rated 2 to 3 times TOTAL: 4.6 hours

4 What can the computer do?

5 What can the computer do? Human Rating 5 minutes Computer Rating 1 minute Essays per Rater hours to rate 500 essays vs. 4.6 hours

6 Motivation and Purpose Low cost options Immediate scoring Improved time management To investigate whether the scores of Criterion can be utilized to help make placement decisions in an ESL program.

7 AWE Validation Studies Ø High level of correspondence e-rater IntelliMetric Intelligent Essay Assessor (IEA) (correlation) 87-97% (Exact agreement) (correlation) 56-88% (Exact agreement) (correlation) Attali & Burstein, 2005; Burstein, Chodorow, & Leacock, 2004 Elliot, 2003; Vantage Learning, 1998, 1999, 2000, 2001 Landauer, Laham, and Foltz (2003)

8 Computer Scoring for Placement Ø Concerns ACCUPLACER OnLine WritePlacer Plus by IntelliMetric Impersonal Distorts the nature of writing Discriminates according to: length grammar mechanics Weak correlations may be due to: lack of formal training/calibrating human evaluators Herrington and Moran, 2006 Jones, 2006 James, 2006

9 Computer Scoring for Placement Ø Validity ACCUPLACER OnLine WritePlacer Plus by IntelliMetric not that much worse.. than placement by readers (p. 126) Useful with spotchecking and retesting a valid tool for assessing writing samples and placing students in composition courses Herrington and Moran, 2006 Jones, 2006 James, 2006

10 Research questions Ø RQ1. What is the relationship between Criterion output and EPT decisions? Holistic scores Trait feedback Ø RQ2. To what extent can holistic scores of Criterion distinguish between different levels of ESL writing classes?

11 Participants Ø 135 international undergraduate students Ø Fall semester 2012 at ISU Disciplines Number of participants Engineering 48 LAS 37 Business 33 Design 10 Human Science 5 Agriculture 2

12 Setting Ø Paper-based English Placement Test (30 min.) Ø Topic: Modern convenience from Criterion Topic category - College level first year Topic mode - Persuasive Modern conveniences such as fast food, automatic teller machines, and labor-saving appliances promise to make life easier. Do these products and services actually make our lives more convenient or do they simply create new problems? Explain your position with reasons and examples from your own experience, observations, or reading.

13 Ø Number of raters Experienced: 9 New: 9 Rating Procedure Ø Rubric based on ACTFL Proficiency Guidelines General Description, Organization, Grammar & Vocab, Functional, Mechanics, and, Comprehensibility Ø Placement based on two raters agreement Third rating for controversial papers Inter-rater reliability: 62% exact agreement

14 EPT Scoring Criteria Advanced Mid (Pass) Advanced Low (101C/D) Intermediate High (101B) ü able to meet a range of work and/or academic writing needs. ü able to narrate and describe with detail in all major time frames ü cohesive devices in texts up to several paragraphs ü good control of the most frequently used target-language syntactic structures and a range of general vocabulary. ü able to meet basic work and/or academic writing needs. ü able to narrate and describe in major time frames ü a limited number of cohesive devices, ü some redundancy and awkward repetition ü some additional effort may be required in the reading of the text. ü able to write compositions and simple summaries related to work and/or school experiences ü inconsistent in the use of appropriate major time markers, resulting in a loss of clarity. ü correspond to those of the spoken language.

15 Curriculum Ø ESL Writing Curriculum Placement Decisions Engl 101B Engl 101C Pass/ Eng150

16 Materials Writing samples from EPT Stratified Random Sampling EPT Level Two-rater Samples Verbatim Transcription Three-rater Samples Word count (M) 101 B C Pass

17 Data Collection Ø Entering essays into Criterion Ø Data extraction Holistic scores Trait feedback (error numbers are normalized) Grammar (S-V agreement, fragment and etc.) Usage (wrong article, preposition errors and etc.) Mechanics (spelling, missing comma, and etc.) Style (repetition of words, short sentences, and etc.)

18 Criterion Scoring rubric 4 3 Slights some parts of the task Treats the topic simplistically or repetitively Is organized adequately, but you need more fully to support your position with discussion, reasons, or examples Shows that you can say what you mean, but you could use language more precisely or vigorously Demonstrates control in terms of grammar, usage, or sentence structure, but you may have some errors. Neglects or misinterprets important parts of the topic or task Lacks focus or is simplistic or confused in interpretation Is not organized or developed carefully from point to point Provides examples without explanation, or generalizations without completely supporting them Uses mostly simple sentences or language that does not serve your meaning Demonstrates errors in grammar, usage, or sentence structure

19 Data Analysis Ø RQ1: Criterion output vs. Human ratings à Descriptive Statistics à Correlation à Regression Ø RQ2: Criterion output differences b/w EPT levels à ANOVA

20 RQ1: Criterion output vs. Human ratings Distribution of Criterion scores over EPT levels B (N=43) C (N=45) Pass (N=44) Criterion scores

21 RQ1: Criterion output vs. Human ratings Ø Correlation (Spearman rho) EPT levels (complete set N=132) EPT levels (two-rater N = 89) EPT levels (three-rater N=43) Criterion scores (N=132) Criterion scores 0.39** 0.47** Ø GUMStyle Word count 0.25** 0.31** ** Total errors -0.40** -0.48** ** Grammar -0.25** -0.33** ** Usage -0.21* ** Mechanics -0.28** -0.30** ** Style -0.30** -0.35** ** ** significant at 0.05

22 RQ1: Criterion output vs. Human ratings Ø Regression analysis (Criterion scores) Model Beta t p-value Constant Word count Total errors Grammar Usage Mechanics Style Dependent variable: Criterion scores R 2 is 0.727

23 RQ1: Criterion output vs. Human ratings Ø Regression analysis (EPT levels) Model Beta (standardized coefficient) t p-value Constant Word count Total errors Grammar Usage Mechanics Style Dependent variable: EPT levels R 2 is 0.208

RQ2: Differences b/w EPT levels Ø Post-hoc Multiple comparison in One-way ANOVA (N=135) Model B-C C-Pass B-Pass Criterion Scores -0.139-0.580* -0.719* Word count -1.489-41.978* -43.

24 RQ2: Differences b/w EPT levels Ø Post-hoc Multiple comparison in One-way ANOVA (N=135) Model B-C C-Pass B-Pass Criterion Scores * * Word count * * Total errors 3.02* * Grammar * Usage * 0.562* Mechanics * 1.180* Style 2.740* * *. The mean difference is significant at 0.05 level

25 Discussion Ø RQ1 à relatively low correlation May be due to: Different grading rubrics Essay lengths Essay prompt level on Criterion (College 1 st year) Ø RQ2 à Distinguished Pass from 101B / C Can - because of: wide coverage of error categories Cannot - because of: style (repetition and spelling)

26 Implications & Future Studies Ø Potential use for distinguishing PASS from Non-Pass confirming placement through diagnostic test Ø Future studies on The effects of different essay topic categories and mode The predictive evidence of Criterion output Paper-based writing vs. computer-based writing

27 References Ø Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater V. 2. The Journal of Technology, Learning and Assessment,4(3). Retrieved from Ø Fulcher, G. (1997). English Language Placement Test: Issues in reliability and validity. Language Testing, 14(2), Ø James, C. L. (2006). Validating a computerized scoring system for assessing writing and placing students in composition courses. Assessing Writing, 11, Ø Ware, P. D., & Warschauer, M. (2006). Electronic feedback and second language writing. In Hyland & F. Hyland (Eds.), Feedback in second language writing: Contexts and issues (pp ). Cambridge: Cambridge University Press.

28 Thank you! Questions and Comment? Acknowledgements: Yoo Ree Chung Zhi Li Hyejin Yang Stephanie Link Volker Hegelheimer Website: volkerh.public.iastate.edu/awe 28

Name Period Date. Grade 9, Unit 3 Pre-assessment. High Stakes for Children in Immigration Reform. By: Alison Burns

Name Period Date. Grade 9, Unit 3 Pre-assessment. High Stakes for Children in Immigration Reform. By: Alison Burns Name Period Date Grade 9, Unit 3 Pre-assessment Please read the following article and answer the questions below: High Stakes for Children in Immigration Reform By: Alison Burns RICHMOND, Va. - As a U.S.