Test Construction and Validity

Theory Basis:  Psychometrics

In the August 2012 Blog Post on Evaluating Training – What it all about? we took a look at a method to determine if the training that was completed was transferred to practical use on the job. This post is going to focus in on the concept of validity as it applies to instructor-developed tests prepared by internal talent.

Validity Defined:

Something is “valid” (has validity) when can actually support the intended point or claim; acceptable as cogent: "a valid conclusion".  

Narrowing Down the Topic

As in other posts I began by mind mapping the topic.  The map grew at a geometric rate and I came to the realization that some of the topics on the mind map could yield a map of their own.  I narrowed down the information to support the content of this blog and came up with the following:

To provide a frame of reference to the topic of this post you can follow the topics in blue text. Beginning with the realm of educational evaluations, we are going to focus on evaluating the individual, specifically the student, and the evaluation that takes place at the conclusion of the instruction; summative.  From the types of summative evaluations, we’ll concentrate on the internal, instructor developed tests that are criterion-referenced tests and how to ensure their quality through being valid.

Clarification

A common misunderstanding of the term criterion is its meaning. Many, if not most, criterion-referenced tests involve a cut-score, where the examinee passes if their score exceeds the cut-score and fails if it does not (often called a mastery test). The criterion is not the cut-score; the criterion is the domain of subject matter that the test is designed to assess.

 

Making the Connection to ISD

For us, the subject matter is the tasks derived in the analysis phase of your chosen design approach.  It is the behavioral repertoire (the kinds of things each examinee can do).  This repertoire is found in the action verbs of the objectives developed from the task identified in the analysis process.

Making our test’s validity cogent comes from two qualities: content validity and construct validity.

Content validity is the extent the items on the test are representative of the domain or universe that they are supposed to represent (the criteria).  To impart content validity to your test you may only as questions related to the objectives.

Construct validity is the extent the test measure the traits, attribute, or mental process it should measure.  This comes from the construction of the test items.  To be valid, a test item’s action verb must be congruent (matching) with the verb in the learning objective.  Beck (1) refers to as “item-objective congruence”; he goes on to say it is “the most important item characteristic.  Graphically it could look like:

How good is good enough?

Is it necessary to test on all the behaviors in a criterion?  In general I would say “Yes”.  For those of us that follow HM-FP-01, Section 3.2, Examination Preparation, Administration, and Control the answer is; “It depends”.  Part 4, Developing Examinations provides guidance:  If your developing items for an item bank all learning objectives will have 3 exam items for each objective. For individual exams, 80% of the learning objectives should be covered.  Based on the above, a note in Step 8 indicates “All objectives should be adequately covered in the exam items.  (I guess you get to define adequately J )

Setting the cut-score is mostly a case of judgment or negotiation. Your best bet is to follow your established standard.  Some standards can be found in the training program descriptions (TPDs) others in company procedures and guides.  If you would like to investigate this topic further I suggest reading Methods for Setting Cut Scores in Criterion-Referenced Achievement Tests a comparative analysis of six recent methods with an application to tests of reading in EFL by Felianka Kaftandjieva.

Challenge

Next time you prepare a test take the time to evaluate your questions for validity by ensuring questions come from the criteria and there is congruence between the objective and test item.

Regards,

Cj

References

Beck R. A. (1982). Criterion-Referenced Measurement: The State of the Art. Baltimore, Maryland: John Hopkins.