The measurement model of student assessment

In the 4th century BC, during the Han Dynasty, education was the means of selecting those who excelled in their study, for they are the ones, so Confucius said, who ‘should become officials’ (Zheng, 1999). The purpose of education was selective, assessment norm-referenced as we say today, its task to do the selecting efficiently. This remained the case in Hong Kong until the late 1990s, and in the minds of many teachers and more parents, it still is, despite the best rhetoric of the Education Commission. Hong Kongers know about banding, and the stern role that what was then the Hong Kong Examinations Authority, now the Hong Kong Assessment and Examinations Authority, played in sorting out the Band 5 geese from Band 1 swans.

However, the obsession with selectivity is not however confined to countries of the Confucian heritage. For most of the last century, assessment systems in schools and universities were designed and implemented to serve selectivity. Who are the high fliers, who the hewers of wood and the drawers of water? Where did Johnny come in class? These were the questions that parents and teachers thought they wanted answered. It was assumed that only a few should obtain high grades, only a few should fail, and most students should fall in between.

Thus, Australian universities use the following grades: Pass, Credit, Distinction, High Distinction—implying that only a few students are to be dubbed ‘highly distinguished’. At least the A B C D system of grading does not carry that implication, although Boards of Examiners get very upset, snorting ‘slack standards’, if most students are awarded A. In my reckoning, if most students in a class do reach the standards deserving an award of A, then we all should be very pleased indeed. But most are not, because allocating grades is not a matter of reaching certain standards, but of measuring the distance between students. ‘A good test gives a good spread’, measurement experts damagingly impressed on teacher education students.

There is a list of topics to teach, with little or no stipulation of how well students should understand them: we simply ‘teach’ them, and when we have done so, we set an exam. In that exam, we’ll probably set a few trick off-the-syllabus questions in order to ensure there are no geese masquerading as swans. Geese don’t study outside the syllabus, you see.

Assessment here has no intrinsic relation to the intended learning outcomes. How could it? The outcomes are unspecified; there are no explicit criteria for the students to meet. The syllabus topics are specified, but they only tell teachers what they have to do; they have to ‘cover’ that list of topics, an unhelpful word if ever there was one, because ‘the greatest enemy of understanding is coverage’, as Howard Gardner (1993: 24) puts it. In the absence of criteria, grading can only be based on how students compare to each other. This isn’t what university teachers really want to know, even if some of us think that it is.

This situation draws on the philosophy and techniques of the measurement model of assessment (Taylor, 1994). The measurement model derives from the study of individual differences. It is aimed at measuring an underlying trait or ability, expressing that measurement along a graduated scale so that individuals can be compared to each other or, as in the case of the IQ for example, to population norms. Complex performances need to be reduced to a unidimensional scale, an essential requirement if comparisons are to be made. Measurement theory can be extremely sophisticated, as anyone who has been involved with the workings of the previous Hong Kong Examinations Authority will know, but the application of measurement theory to education is ‘twentieth century statistics applied to nineteenth century psychology’ (Pellegrino, Baxter and Glaser, 1999). Some educators like it, because it gives the impression of being scientific, objective and precise, but it is not. It is the wrong model to apply to classroom assessment.

The essence of the measurement model is quantification. Educational outcomes are usually quantified either by breaking knowledge into binary units, such as words, ideas, points that are either correct or incorrect, or by arbitrarily employing ‘marks’ and converting them to percentages. In the first, each correct unit is considered to be worth the same as any other unit, an essential requirement if they are to be added or averaged. The final performance is then assessed as the sum of the marks awarded.

It doesn’t matter what the student gets correct, as long as there are (usually) fifty of them. The integrity of the performance has been lost.

Imagine if Oscars were awarded by marking each frame of a movie out of ten, and then counting up to see which movie got most marks. Do that and you lose the plot. Just so do analytic summative assessments in educational institutions lose the academic plot. Giving students feedback on how well they are doing on component aspects of a task is essential formatively, but the final summative evaluation of a complex performance makes sense only when it seen as a whole.

If analytic marking in component scores doesn’t make epistemological sense, adding and averaging those scores doesn’t make educational sense. Consider. A high average in most components of a course, but a failure in one component, means the student passes. The reasoning here is straight from the measurement model: that student clearly had the ability to pass, so an overall pass is justified. The fact that the student happened not to have passed one important component is brushed aside. Surely the question is not how able the student is, but how well the content to be learned was actually learned. The focus should not be on the person, but on the performance. The logic of awarding a pass to a student on a section of the course in which that student has already failed is difficult to grasp, but it is an arithmetical consequence of quantitative marking procedures. It’s like saying surgery students can pass if they get enough marks in anaesthetics and skilful use of the scalpel to cover a failure in anatomy, which might result in removing the wrong part.

It is assumed that percentages, whether created by the proportion of correct responses, or by using rating scales that add up to 100, create a universal currency that is equivalent both across subjects areas and across students. Universities have long and earnest debates about one faculty using 75% as the cutoff for an Agrade, and another using 70% as the cutoff. Such debates are trying to extract certainty from the unknowable. There is simply no way of knowing if 75% in one subject is ‘the same standard’ as 75% in another.

The measurement model requires that a test spreads students out, clearly sorting the high from the low performers, most often according to the normal curve on the assumption that ability determines the outcomes of learning—as ability is normally distributed, near enough, so should our grades be normally distributed. But wait a minute. Shouldn’t effective teaching reduce the variance between students rather than increase it? Shouldn’t good teaching lift the performance of a hitherto poor performing student nearer to that of a good student? I think so, but teachers who produce results at odds with the normal distribution are criticized by their peers as being either too lenient or too harsh, usually the former.

The measurement model also assumes that what is being measured is stable over time. If you are selecting students for academic ability, such as selection for the Imperial Court, or for graduate school, you need to assume that their relative ability will not easily change over time. Thus, tests should not only be unidimensional but have a high test-retest reliability. Yet we know that teaching is supposed to produce change in what is being measured, a change usually called ‘learning’.

Such tasks and standardized conditions may not allow many students to display their best learning. Individuals learn and perform optimally under different conditions. Some work best under pressure, others need more time; some best make their points in writing, others visually or orally. Many need opportunities for reflection and revision if they are to show their best.

Such considerations bring a different meaning to the word ‘fairness’. Isn’t fairness allowing each student to develop his or her potential? Is education a level playing field or a garden of opportunity?

The backwash from the effects of quantitative assessments on teaching and learning are negative (Elton, 1987; Crooks, 1988; Frederikson & Collins, 1989). Such testing tells students:

. Ideas are equally important

. You can skip or slack on certain areas if you are doing well elsewhere

. Declarative knowledge, out of context learning, is more important than application

. Trees are more important than the wood

. Verbatim responses must gain marks

. Success and failure depend largely on uncontrollable factors such as relative ability and luck

. Assessment is the responsibility of the teacher, selfassessment or reflection are unimportant

Students in their search for marks may easily fail to see the structures being learned; in counting the trees, they get lost in the wood. Disputes about grades become a niggling quibble about an extra mark here an extra mark there. This is demeaning for both student and teacher.

As the final grade depends on relative performance, students can easily see that their success or failure depends on luck: who in the class happened to be cleverer than they. Finally, assessment in the measurement model is completely in the teacher’s hands. This should not be so; one of the important things that students need to learn as practicing professionals is how to assess their own work.

مواضيع ذات صلة

The SOLO taxonomy The theoretical background of the SOLO taxonomy

Views on the Adoption and Implementation of the SOLO Taxonomy Introduction

Constructive alignment

The standards model of assessment

Assessment by portfolio: an experiment

Assessment

Teaching and learning activities Monitoring and assessment