Exploring effect of rater on prediction error in automatic text grading for open-ended question

Che Di Lee*, Chun-Yen Chang, Tsai Yen Li, Hsieh-Hai Fu, Tsung Hau Jen, Kang Che Lee

*此作品的通信作者

研究成果: 書貢獻/報告類型會議論文篇章

摘要

This paper aims to explore the way of evaluating the automatic text grader for open-ended questions by considering the relationships among raters, grade levels, and prediction errors. The open-ended question in this study was about aurora and required knowledge of earth science and physics. Each student's response was graded from 0 to 10 points by three raters. The automatic grading systems were designed as support-vector-machine regression models with linear, quadratic, and RBF kernel respectively. The three kinds of regression models were separately trained through grades by three human raters and the average grades. The preliminary evaluation with 391 students' data shows results as the following: (1) The higher the grade-level is, the larger the prediction error is. (2) The ranks of prediction errors of human-rater-trained models at three grade levels are different. (3) The model trained through the average grades has the best performance at all three grade-levels no matter what the kind of kernel is. These results suggest that examining the prediction errors of models in detail on different grade-levels is worthwhile for finding the best matching between raters' grades and models.

原文英語
主出版物標題Proceedings of the 17th International Conference on Computers in Education, ICCE 2009
頁面462-466
頁數5
出版狀態已發佈 - 2009
事件17th International Conference on Computers in Education, ICCE 2009 - Hong Kong, 香港
持續時間: 2009 十一月 302009 十二月 4

出版系列

名字Proceedings of the 17th International Conference on Computers in Education, ICCE 2009

其他

其他17th International Conference on Computers in Education, ICCE 2009
國家/地區香港
城市Hong Kong
期間2009/11/302009/12/04

ASJC Scopus subject areas

  • 電腦科學(雜項)
  • 教育

指紋

深入研究「Exploring effect of rater on prediction error in automatic text grading for open-ended question」主題。共同形成了獨特的指紋。

引用此