The Pentavirate

If you’re at all prone to conspiracy theory, avoid reading this interview with John Popham, an educator and former standardized-test writer, in which he gives some awfully sobering facts about standardized tests:
A nationally standardized achievement test is given in about an hour. In about an hour, you can't test all that much, so you have to sample from larger domains of knowledge and skills. And what you end up with sometimes does not match at all well with what's being taught in school or what's supposed to be taught in school. Some studies suggest that fully 75 percent of what is on a test is not even supposed to be covered in a particular school. Clearly, it's unfair to judge the quality of schooling based on a test that's largely covering things that ought not be taught.


If one compares the content of textbooks used in mathematics with standardized achievement tests in mathematics, you will frequently find that fully half of the content in the test is not addressed in those textbooks.

So the tests aren't assessing retention of the facts and concepts we teach in class?  What are they intended to do, then?
You want to have a very substantial spread of scores. And one of the best ways to do that is to have questions that are answered correctly by about 50 percent of the kids; 50 percent get it right, 50 percent get it wrong. You don't want items in there that are answered by large numbers of youngsters: 80 percent, 90 percent. Unfortunately, those items typically cover the content the teachers thought important enough to stress.

So the more significant the content, the more the teacher bangs at it, the better the kids do. And as soon as the kids do very well in that item, 80 percent, 90 percent getting it right, the item will be removed from the test. ... So you miss items covering the most important things that teachers teach. ...

The rest of the interview is just as troubling: he mentions, among other things, that the cheapest test-scoring option (multiple choice) is the most frequently used, even though more expensive options (written and performance-based responses) are far better at measuring the nuances of a student's knowledge. Overall, the piece does give an ominous feeling of behind-the-scenes collaboration, the kind designed to make educators and policy-writers look good at the expense of struggling students.

The only encouragement came from Popham's own opinions about how to write tests.  Here, the interviewer asked him how he would go about creating a fair assessment:
I'd go to a specialist and I'd say, "Isolate the things that you want children to be able to do and put them in three piles: the absolutely essential, the highly desirable and the desirable." And having done that, then I get those two piles away and just go with the absolutely essential. And then I would say, "Now rank them from top to bottom; the most important, the next most important," and so on.

And then I would have the assessment people come in and say, "These four can be assessed in the time we have available, and can be assessed in such a way that teachers will know how to promote children's mastery of them."

Advice worth taking for any teacher who writes a test.  I like that: separate the essential from the desirable, and figure out how to assess knowledge.  Sounds simple enough, but I'm guessing it will take a lifetime to even come close.