Proposing ways of evaluating automatic short-answer markers with multiraters

Che Di Lee, Tsung Hau Jen, Hsieh Hai Fu, Chun Yen Chang

研究成果: 雜誌貢獻文章

1 引文 斯高帕斯(Scopus)

摘要

A method of evaluating automatic short answer markers (ASAM) with multiraters has been proposed. Three indexes including mean prediction bias (MPB), prediction-bias change with scores (PBCS) and PBCD have been suggested to analyze systems' performance in detail. The first is to look at the direction of bias instead of only the size of error. The second is to look at the system performance at each score instead of only overall performance. The third is to look at the relationship between the system performance and the rating deviation instead of wasting the information provided by the average scores. The fourth is to look at the performance sensitivity by regression analysis instead of only employing qualitative analysis. Moreover, the evaluation points that the first priority of improving our single-word based system is to decrease the prediction error at low and high scores. The analysis reveals that many low- and high-score responses are misclassified as middle scores.

原文英語
期刊British Journal of Educational Technology
43
發行號3
DOIs
出版狀態已發佈 - 2012 五月 1

ASJC Scopus subject areas

  • Education

指紋 深入研究「Proposing ways of evaluating automatic short-answer markers with multiraters」主題。共同形成了獨特的指紋。

  • 引用此