Assessing creative problem-solving with automated text grading

Hao Chuan Wang, Chun Yen Chang, Tsai Yen Li

Research output: Contribution to journalArticle

51 Citations (Scopus)

Abstract

The work aims to improve the assessment of creative problem-solving in science education by employing language technologies and computational-statistical machine learning methods to grade students' natural language responses automatically. To evaluate constructs like creative problem-solving with validity, open-ended questions that elicit students' constructed responses are beneficial. But the high cost required in manually grading constructed responses could become an obstacle in applying open-ended questions. In this study, automated grading schemes have been developed and evaluated in the context of secondary Earth science education. Empirical evaluations revealed that the automated grading schemes may reliably identify domain concepts embedded in students' natural language responses with satisfactory inter-coder agreement against human coding in two sub-tasks of the test (Cohen's Kappa = .65-.72). And when a single holistic score was computed for each student, machine-generated scores achieved high inter-rater reliability against human grading (Pearson's r = .92). The reliable performance in automatic concept identification and numeric grading demonstrates the potential of using automated grading to support the use of open-ended questions in science assessments and enable new technologies for science learning.

Original languageEnglish
Pages (from-to)1450-1466
Number of pages17
JournalComputers and Education
Volume51
Issue number4
DOIs
Publication statusPublished - 2008 Dec 1

Keywords

  • Automated grading
  • Computer-aided assessment
  • Creative problem-solving
  • Machine learning application
  • Science learning assessment

ASJC Scopus subject areas

  • Computer Science(all)
  • Education

Fingerprint Dive into the research topics of 'Assessing creative problem-solving with automated text grading'. Together they form a unique fingerprint.

  • Cite this