Should we be concerned about validity...?

Validity theory has, since the mid '50s, been based on the Standards for Psychological and Educational Testing, jointly developed by AERA, APA, and NCME. Until the '80s, the vast majority of tests were norm-referenced, and techniques for content, construct,and some types of criterion-referenced validation (predictive and concurrent, for example) involved precise mathematical computations. Before competency-based assessment began to gain popularity, most administrators of assessment systems were not concerned with validity, something which had always been left to the statisticians and test developers. Without a background in research or measurement, most administrators are not even aware that validity threats exist, instead believing (if subconsciously) that anything that comes out of a computer must be true.

The authors of the Standards, however,describe six types of validity, and contend that if ANY identified threats are not addressed, the accuracy of the scores comes into question. We are most familiar with examining the reliability of scoring practices among assessor. The literature suggests that there is more scoring error introduced by differences in assignments than by assessors' scoring. Yet the actual activities that are meant to engage students in demonstrating performance standards come under little scrutiny.

Do we have an obligation to educate ourselves about validity issues?  Where do we find the time and motivation to investigate the "Black Box" called validity?