Linguistic feature analysis of cefr labeling reliability and validity in language textbooks

Despite the importance of grading language textbooks for teaching and learning, few studies have addressed the issues of reliability, validity, and efficiency of grading texts. This study adopted an automated textbook grading system to examine the grading consistency of five L2 Chinese textbook series labeled with CEFR difficulty levels. Twelve linguistic features were selected to represent the most crucial aspects of text readability: lexicon, semantics, syntax, and cohesion. Both the validity and reliability of grading assignments were tested between and within textbook series. The results suggested that 4 out of the selected 5 textbook series did not assign grading levels accurately reflective of actual text difficulty.