To “measure” something in an individual’s mind is a difficult task. Take, for example, someone’s knowledge of a specific science concept such as energy transformations. How would you go about quantitatively characterizing someone’s knowledge of energy transformations? Where would you start? It’s a tough question to answer. These are the things I was thinking about on today’s run.
Let me suggest two different ways to go about such as task. The first way is to think up or find some questions about energy transformations and assemble them into a test. You would probably pick or write some great questions which span a range of difficulty, and you would consult literature in the field and even experts to provide evidence that the questions were not only conceptually accurate, but also valid. You would then administer these questions to the individual, calculate some score, compare it to a previously-defined cut score or normative sample and make some inferences about the individual’s level of knowledge. All fine and good. But this approach may be too simple.
Now ask yourself what you really mean by “knowledge of energy transformations.” It is likely that what you will describe is much more nuanced than what is captured in the set of items you put together in the above example. You will probably generate some theory (small “t” theory) about what different levels of sophistication of this knowledge would look like. Maybe individuals at the highest level will exhibit an integration of concepts about energy with concepts about forces. Moreover, they may articulate these understandings when rationalizing their answers to “energy” questions. This theory about what that sophistication of understanding looks like is what we call a construct. And this construct exists in the mind of the individual, although it is latent. In order to obtain a measure of this latent construct in the individual, you need an instrument (i.e., set of items) which relates closely to, and in fact is defined by, that construct. Further, scores from that instrument must be able to be mapped back onto that construct. The key then is to identify what qualitatively distinct level of that construct where an individuals lies.
Did your set of items gathered in the above approach give you enough information to make the same types of inferences? Probably not. This construct-based approach to instrument development and obtaining measures is much more difficult, but also provides a much more solid foundation for making inferences about someone’s ability or knowledge. But it takes time to engage in this second method. Lots of time. And expertise. So it is not always the way to go. But in those cases where the stakes are high, such as assessing the efficacy of an expensive treatment, construct-based measures are warranted (even necessary). Another challenge is explaining to those not familiar with this method why you would choose to undertake such a task. But engaging in developing a construct-based instrument and measure are sure to do one thing: help you to further know that which you are measuring.
Today’s run was pretty good, and supported some good thinking.
*a good place to start if you’d like to learn more about the construct-based approach to instrument development is this book by Mark Wilson:
Wilson, M. (2005). Constructing Measures: An item response modeling approach. Erlbaum, Mahwah, NJ. http://goo.gl/pX5f9
