Exploring effect of rater on prediction error in automatic text grading for open-ended question

Che Di Lee, Chun-Yen Chang, Tsai Yen Li, Hsieh-Hai Fu, Tsung Hau Jen, Kang Che Lee

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This paper aims to explore the way of evaluating the automatic text grader for open-ended questions by considering the relationships among raters, grade levels, and prediction errors. The open-ended question in this study was about aurora and required knowledge of earth science and physics. Each student's response was graded from 0 to 10 points by three raters. The automatic grading systems were designed as support-vector-machine regression models with linear, quadratic, and RBF kernel respectively. The three kinds of regression models were separately trained through grades by three human raters and the average grades. The preliminary evaluation with 391 students' data shows results as the following: (1) The higher the grade-level is, the larger the prediction error is. (2) The ranks of prediction errors of human-rater-trained models at three grade levels are different. (3) The model trained through the average grades has the best performance at all three grade-levels no matter what the kind of kernel is. These results suggest that examining the prediction errors of models in detail on different grade-levels is worthwhile for finding the best matching between raters' grades and models.

Original languageEnglish
Title of host publicationProceedings of the 17th International Conference on Computers in Education, ICCE 2009
Pages462-466
Number of pages5
Publication statusPublished - 2009 Dec 1
Event17th International Conference on Computers in Education, ICCE 2009 - Hong Kong, Hong Kong
Duration: 2009 Nov 302009 Dec 4

Publication series

NameProceedings of the 17th International Conference on Computers in Education, ICCE 2009

Other

Other17th International Conference on Computers in Education, ICCE 2009
CountryHong Kong
CityHong Kong
Period09/11/3009/12/4

    Fingerprint

Keywords

  • Automatic grader
  • Prediction error
  • Rater
  • SVM
  • Science learning
  • Testing

ASJC Scopus subject areas

  • Computer Science (miscellaneous)
  • Education

Cite this

Lee, C. D., Chang, C-Y., Li, T. Y., Fu, H-H., Jen, T. H., & Lee, K. C. (2009). Exploring effect of rater on prediction error in automatic text grading for open-ended question. In Proceedings of the 17th International Conference on Computers in Education, ICCE 2009 (pp. 462-466). (Proceedings of the 17th International Conference on Computers in Education, ICCE 2009).