Slide 1 : 1 Mohd Saidfudin Masodi
IRCA (Lon.) Lead Assessor
Research Fellow,
Center of Engineering EducationUniversity Kebangsaan Malaysia Why Rasch ?
How to convince your friends NOT to misuse raw scores : September 23, 1999 COMET 2 How to convince your friends NOT to misuse raw scores Benjamin D. Wright
Institute for Objective Measurement
& MESA Psychometric Laboratory
bd-wright@uchicago.edu
THE TROUBLE WITH RAW SCORES : September 23, 1999 COMET 3 THE TROUBLE WITH RAW SCORES The CRUMBLING CATEGORIES of Lumpy Ratings
The RUBBER RULER of Irregular Intervals
The PERVERSE PRECISION of Extreme Scores: Squashed Extremes
The BROKEN BUCKET of Missing Data
The DIRTY DATA of Unpredictable Responses
The CRUMBLING CATEGORIES of Lumpy Ratings : September 23, 1999 COMET 4 The CRUMBLING CATEGORIES of Lumpy Ratings Rating Forms Offer Equally Spaced Categories
1 . 2 . 3 . 4 . 5 . 6 . 7
But Raters Reply with Unequally Spaced Responses
1 . . . 2 . . 3 4 . 5 . . 6 . . . . . 7
The Measure Distance of One More Point from Category 1 to 2 can be FOUR times BIGGER than The Measure Distance of One More Point from Category 3 to 4 !!
Q.1 Are you sure you want to mistake Lumpy Ratings for Equal Interval Measures?
RATING SCALE: UNSTABLE INTERVAL : RATING SCALE: UNSTABLE INTERVAL
RATING SCALE: UNSTABLE INTERVAL : RATING SCALE: UNSTABLE INTERVAL Item 1 Item 2 median 1 median 2 Q.1 Are you sure you want to mistake Lumpy Ratings for Equal Interval Measures?
The RUBBER RULER of Irregular Intervals and Squashed Extremes : September 23, 1999 COMET 7 The RUBBER RULER of Irregular Intervals and Squashed Extremes When our items bunch in clumps of equally difficult items then a count of one more right answer within a clump implies only a little increase in our ability.
But when we leap ahead and our next right answer is in a distinctly harder clump, then we see that this one more right implies a large increase in our ability.
As for the ends of the test where one more right is from 0 to 1 or from all but one to all.
Then the implied change in our ability is infinite.
Slide 8 : September 23, 1999 COMET 8
The PERVERSE PRECISION of Extreme Scores : September 23, 1999 COMET 9 The PERVERSE PRECISION of Extreme Scores The Statistical Precision of a Raw Score is MAXIMUM at exactly the place where the Information a Raw Score provides is MINIMUM; 10% correct at 0.5 and 2.5logit difference.
When a person gets the lowest possible score, their raw score precision is perfect. We know exactly the score their low ability implies. But we have no idea how far Below that Score their ability might be!
Similarly, the highest possible score is 100marks but we have no idea how far Above that Score their ability might be!
They are Off-Our-Scale and Our Precision for their Unknown Measure is ZERO!
Q.2 Are you sure you want to mistake imprecise raw scores for precise (and additive) measures?
The BROKEN BUCKET of Missing Data : September 23, 1999 COMET 10 The BROKEN BUCKET of Missing Data Compare Two Persons on an 8-item test, 7-category Functional Independence Measure:
Person A: 2 3 3 m m m m 4 = 12
Person B: 1 2 2 2 2 2 2 3 = 16
Which Patient is more Able?
Person B appears to have a higher score 16 > 12 on all 8 items, BUT on the 4 Items A and B have in common
Person A: 2 3 3 m m m m 4 = 12
Person B: 1 2 2 m m m m 3 = 8
Now, Person A has the higher score of 12 > 8 ; ??
Q.3 Are you sure you want to misuse missing-data-leaking raw scores for missing-data-impervious measures?
The DIRTY DATA of Unpredictable Responses : September 23, 1999 COMET 11 The DIRTY DATA of Unpredictable Responses When Item Responses are arranged from Easy items to Hard items you expect a response patterns like:
7 7 6 6 6 5 5 4 = 46
4 4 3 3 2 1 1 1 = 19
BUT suppose you get:
7 7 6 6 {1} 5 5 4 = 41 ?
4 4 3 3 {7} 1 1 1 = 24 ?; then WHAT ?
Raw Scores are BLIND to Unpredictable Responses.
Only Quality Control of Well-Constructed Measures Tells you about Response Surprises
Q.4 Are you sure you want to suffer raw-score-dirty-data blindness instead of enjoying data-vigilant measures?
WHAT ARE VARIABLES? : September 23, 1999 COMET 12 WHAT ARE VARIABLES? Length and weight may be real variables. But we construct their units of measure.
Inches and ounces are our creations - Our own imaginative constructions.
A variable is an amount of something which we can always picture as a distance
>From Less -------------------------> To More
We can arrange to experience evidence of this "something".
BUT1, its (linear) measurement line and its units of measurement are up to us to construct.
EVIDENCE OF A VARIABLE? : September 23, 1999 COMET 13 EVIDENCE OF A VARIABLE? The variable and its evidence could be:
length benchmarks exceeded
health symptoms absent
ability problems solved
skill tasks completed
attitude assertions condoned
We can arrange to provoke occurrences of evidence and count how many pieces occur.
BUT2 these counts are not measures.
REQUIREMENTS FOR MEASUREMENT : September 23, 1999 COMET 14 REQUIREMENTS FOR MEASUREMENT Pieces of evidence must be concrete to be observed.
This necessary reality keeps them uneven in size.
To measure we need an even abstraction, a line marked out in abstractly equal units.
Pieces of evidence are unstable. They appear and disappear by accident. They are only probable signs of the variable which they are designed to manifest.
To measure, we must find a way to connect the pieces of evidence we can arrange to observe to the probabilities of the measures we want.
WHAT IS MEASUREMENT? : September 23, 1999 COMET 15 WHAT IS MEASUREMENT? DISTANCE (Length) was our First Variable
COUNTING Steps and Fingers was our First Measuring Operation
The Trouble with Counting is its UNEQUAL UNITS
How many apples fill a basket? How many oranges squeeze a glass? You may not believe it. But can we mix apples and oranges?
YES, we do it all of the time, by WEIGHING them!
CONSTRUCTING MEASURES : September 23, 1999 COMET 16 CONSTRUCTING MEASURES But weighing is a constructed abstraction. There are NO tangible equal units. We have to invent them.
Equal feet are abstracted from real feet. Equal pounds are abstract real weights
We construct our instrumentation of the variables: length and weight to approximate units equal enough (sufficient) to serve our practical purposes
Let’s see a thermometer scale construct…
HOW DO WE HANDLE UNCERTAINTY? : September 23, 1999 COMET 17 HOW DO WE HANDLE UNCERTAINTY? We measure so that we can use the past to plan and navigate the future BUT3 the future is by definition UNCERTAIN
Imagine two batters: Smith bats 400 and Jones bats 200
So which one will hit at their next batter-up? No way to know ahead of time. Even Smith has only a 4 out of 10 record. We can’t wait to find out.
So which one shall we send to the plate? Smith's odds for a hit are 2/3; Jones' odds for a hit are only ¼. Smith odds for a hit are 8/3 times better than Jones'.
Even though we know nothing for sure; Q4..Does any doubt remain as to who to send to bat ?
That's how we handle uncertainty. We use past experience to estimate PROBABILITIES and use these probabilities to foresee the future…
COUNTING ABSTRACT UNITS : September 23, 1999 COMET 18 COUNTING ABSTRACT UNITS To finish this job we have to construct a reproducible transition from counting concrete events, like right answers, observed or reported symptoms, relative agreements, frequency or importance categories to counting abstract units of equal size and wide generasability.
How can we do this?
INVERSE PROBABILITY : September 23, 1999 COMET 19 INVERSE PROBABILITY To deal with the uncertainty we ask Bernoulli, Bayes and Laplace and interpret our observation X as evidence of its occurence probability Px
To construct unit equality we ask Campbell, Thurstone, Rasch and Luce & Tukey and define Px to satisfy the equation: log[Pnix/(1-Pnix)] = Bn - Di
Pnix is the probability of a successful response Xni being produced by Person n to item i
Bn is the ability of Person n
Di is the difficulty of item i
The construction of equal size and hence additive units is called CONJOINT ADDITIVITY.
CLOSING THE DEAL : September 23, 1999 COMET 20 CLOSING THE DEAL Raw scores cause problems with:
lumpy ratings
irregular intervals and squashed extreme score
missing data
unpredictable responses
Rasch Measurement Model provides a solution to these problems by:
using probabilities to predict futures.
providing the ruler with equal size interval and units of measurement… ( I’l show you next )
Slide 21 : 21 Why not SPSS ?
TYPES OF DATAand the mathematical limitations : 22 TYPES OF DATAand the mathematical limitations QUALITATIVE DATA QUANTITATIVE DATA
Slide 23 : 23 Mohd Saidfudin Masodi
IRCA (Lon.) Lead Assessor
Research Fellow,
Center of Engineering EducationUniversity Kebangsaan Malaysia TQ for changing your paradigm …