Assessing creative problem-solving with automated text grading

Hao Chuan Wang*, Chun Yen Chang, Tsai Yen Li

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

74 Citations (Scopus)


The work aims to improve the assessment of creative problem-solving in science education by employing language technologies and computational-statistical machine learning methods to grade students' natural language responses automatically. To evaluate constructs like creative problem-solving with validity, open-ended questions that elicit students' constructed responses are beneficial. But the high cost required in manually grading constructed responses could become an obstacle in applying open-ended questions. In this study, automated grading schemes have been developed and evaluated in the context of secondary Earth science education. Empirical evaluations revealed that the automated grading schemes may reliably identify domain concepts embedded in students' natural language responses with satisfactory inter-coder agreement against human coding in two sub-tasks of the test (Cohen's Kappa = .65-.72). And when a single holistic score was computed for each student, machine-generated scores achieved high inter-rater reliability against human grading (Pearson's r = .92). The reliable performance in automatic concept identification and numeric grading demonstrates the potential of using automated grading to support the use of open-ended questions in science assessments and enable new technologies for science learning.

Original languageEnglish
Pages (from-to)1450-1466
Number of pages17
JournalComputers and Education
Issue number4
Publication statusPublished - 2008 Dec


  • Automated grading
  • Computer-aided assessment
  • Creative problem-solving
  • Machine learning application
  • Science learning assessment

ASJC Scopus subject areas

  • General Computer Science
  • Education


Dive into the research topics of 'Assessing creative problem-solving with automated text grading'. Together they form a unique fingerprint.

Cite this