Examining Uncertainty and Misspecification of Attributes in Cognitive Diagnostic Models

About this book

In recent years, cognitive diagnostic models (CDMs) have been widely used in educational assessment to provide a diagnostic profile (mastery/non-mastery) analysis for examinees, which gives insights into learning and teaching. However, there is often uncertainty about the specification of the Q-matrix that is required for CDMs, given that it is based on expert judgment. The current study uses a Bayesian approach to examine recovery of Q-matrix elements in the presence of uncertainty about some elements. The first simulation examined the situation where there is complete uncertainty about whether or not an attribute is required, when in fact it is required. The simulation results showed that recovery was generally excellent. However, recovery broke down when other elements of the Q-matrix were misspecified. Further simulations showed that, if one has some information about the attributes for a few items, then recovery improves considerably, but this also depends on how many other elements are misspecified. A second set of simulations examined the situation where uncertain Q-matrix elements were scattered throughout the Q-matrix. Recovery was generally excellent, even when some other elements were misspecified. A third set of simulations showed that using more informative priors did not uniformly improve recovery. An application of the approach to data from TIMSS (2007) suggested some alternative Q-matrices.