The Assessment Blog

Should we be concerned about validity...?

2011-01-22T03:10:00.000-08:00

Validity theory has, since the mid '50s, been based on the Standards for Psychological and Educational Testing, jointly developed by AERA, APA, and NCME. Until the '80s, the vast majority of tests were norm-referenced, and techniques for content, construct,and some types of criterion-referenced validation (predictive and concurrent, for example) involved precise mathematical computations. Before competency-based assessment began to gain popularity, most administrators of assessment systems were not concerned with validity, something which had always been left to the statisticians and test developers. Without a background in research or measurement, most administrators are not even aware that validity threats exist, instead believing (if subconsciously) that anything that comes out of a computer must be true.

The authors of the Standards, however,describe six types of validity, and contend that if ANY identified threats are not addressed, the accuracy of the scores comes into question. We are most familiar with examining the reliability of scoring practices among assessor. The literature suggests that there is more scoring error introduced by differences in assignments than by assessors' scoring. Yet the actual activities that are meant to engage students in demonstrating performance standards come under little scrutiny.

Do we have an obligation to educate ourselves about validity issues? Where do we find the time and motivation to investigate the "Black Box" called validity?

Validity Issues ... #4

2010-05-05T03:55:00.000-07:00

4. 'How likely is it that the faculty would be willing to score the work of students other than their own?'.

When grading students in your own courses, you as the course instructor are the most likely person to generate the scores and final grades. Unlike grading an assignment or generating a final course grade, competency-based performance assessment involves the comparison of student work to benchmark indicators or standards of some sort. When instructors evaluate the work of their own students, they may have a tendency to score higher or lower if they know the student personally, based on prior knowledge of the student's general abilities. Assessor bias threatens the accuracy of scores. Please provide comments about how assessor bias is or is not addressed in your assessment system...

Validity Issues... #3

2010-05-05T03:35:00.000-07:00

3. 'How important is it to your assessment team that students and faculty develop advanced eportfolio development software skills?'.

The eportfolio has been called a new genre for the 21st Century. Some scholars believe that eportfolio authoring involves a distinct set of skills and unique conventions, and that eportfolio authoring can be an effective learning tool to improve learner outcomes.

To be useful as a learning tool, eportfolio development must be modeled extensively in the classroom, yet few faculty ever develop an eportfolio collection or incorporate eportfolio development learning activities into the curriculum. Eportfolios are most often seen as a technology enhancement or add-on to traditional education.

Please comment on the use of eportfolio authoring and development in your assessment system...

Validity Issues ... #2

2010-05-05T03:29:00.000-07:00

2 . 'How important is it to your assessment team that each scoring criterion is linked to a single standard within a standard set?'

Each scoring criterion should be a measurement of only one standard. If you give a single score for two standards, you are not getting a true picture of student ability for either standard. Please comment...

Validity Issues in Web-Based Performance Assessment

2010-05-05T03:21:00.000-07:00

1. Regarding the general topic of validity in web-based performance assessment, what , if any, benefits do you perceive there will be to your system and/or the people involved if the validity could be improved? In other words, why worry about the accuracy of scores at all?

What do we want to know about our students?

2010-04-13T03:01:00.000-07:00

So, we are going to assess our students. There's a lot happening in our programs. We can collect lots and lots of data. How much data do we need? It takes time to design an assessment plan. It takes time to assess and more time to produce and analyze the reports. What is the best approach to this complicated task?

What do we really want to know about students? There are basically three things that we want to know...Are our graduates qualified for the certification we have given them? Did they learn what they know and can do from OUR instructional programs? Is our instruction effective for ALL students?

Let's look at the first question: Are our graduates qualified ...? What data do we need and which report do we produce to find this out?

This question is asking about a frequency distribution: a bar chart is the most common report of this type. Once we have produced a bar chart, we will be able to see whether or not our graduates are all scoring at or above the acceptable level. We can produce multiple bar charts for each standard in our learning domains, or we can produce one bar chart with the overall average for all our students, for one or more groups, departments, or programs. That means we can say with confidence that our graduates are qualified.

Did they learn what they know and can do from OUR instructional programs?
...or did they arrive on campus already knowing what they know and with the same skills and abilities as they have now? In order to answer this question, we need to know what our students know and can do BEFORE they start the program. Then we need to be able to compare where they start with where they are at graduation (or any other point in time)...and we need to be able to intepret the difference in these two scores. Is it likely that our instruction influenced these scores so that the student progressed in their understanding because of our instruction? Or, was this learning progression just a natural by-product of time passing and the maturing process? In other words, was the change significant enough that we could pat ourselves on the back...or do we need to go back to the drawing board?

(Answering this question is a little more complicated than producing bar charts...but doable, if we plan ahead.)

Is our instruction effective for ALL students?

...because if it isn't, we're not being FAIR to those students who are not learning the skills that they need for the future. As educators, we know that all students can learn, but that each learner has their own timeline for learning and their own learning preferences. Learners engages with new ideas in different ways and they understand new ideas using different processes and strategies. So, just for starters, there are at least three categories of learner that we need to examine to make sure that we are facilitating learning for each of these groups: How do our students engage with new ideas? How do our students understand new ideas? How do we accomodate various timelines?

(More questions! Oh, boy...this is getting more complicated...!!)