Calibrating the effectiveness of a question paper evaluation tool using automatic keyword extraction

Abstract

Well aligned curricula in every subject of a course place an emphasis not only on the curricular goals but also on the evaluation, teaching and learning activities. Performance evaluation of students in every subject of a course is done by university based examinations with subject specific question paper. Question paper gives proportionate marks to different content areas/topics and include analytical, objective, long and short answer type questions. We address the problem of evaluating question papers at undergraduate level examinations by analysing each question of a question paper on different criteria. University specified syllabus file of a subject along with Bloom’s Taxonomy concept is used as a guideline in evaluating difficulty level of an examination question paper. Using “Blooming” of an examination –that is, identifying its constituent parts on Bloom’s taxonomy, the difficulty level of an examination paper can be pseudo-objectively assessed. Bloom’s Index assist in identifying the proportion of marks allocated to higher order cognitive skills. Text pre-processing and information extraction techniques are used to extract keywords from textual contents in the syllabus file as well as in the question paper. A tool named KEQPBS (Keyword Extraction of Question Paper Based on Syllabus) has been implemented. The tool can be used by the subject expert or question paper setter or question paper moderator or chairperson as a guideline to evaluate or revise the examination paper accordingly.

Keywords: Bloom’s taxonomy, educational data, text pre-processing, pruning, keyword extraction.