What does that cut score mean? - Printable Version
+- WGU Students (http://wgustudents.com/forum)
+-- Forum: Welcome (/forumdisplay.php?fid=1)
+--- Forum: WGU Student Hall (/forumdisplay.php?fid=3)
+--- Thread: What does that cut score mean? (/showthread.php?tid=1704)
What does that cut score mean? - Dana - 05-02-2012 05:27 AM
Thank you Mike, for the info and Mary for the suggestion of making it a sticky:
Objective Assessments: Setting Cut Scores
What is the procedure for setting cut scores?
Have you ever suspected that having 70% equivalent to a C, 80% equivalent to a B, and 90% equivalent to an A is not fair? Some tests are harder than others. Therefore, setting a rigid standard of this sort is usually not fair.
Test questions (items) on a given topic can be written at many levels of difficulty. For example, a subject matter expert could write a question on the U.S. Constitution that is so easy anybody would get it right. Conversely, the subject matter expert can write another question on the same topic on a different assessment that even Supreme Court justices would have trouble answering correctly.
Setting the cut (passing) score for each assessment at 70% in both of these cases would be unfair because one assessment has harder questions. Therefore, WGU needs a way to determine how hard the test items really are and a process for computing the proper cut score from these test item difficulties. This is the purpose of the Angoff procedure.
What is the modified Angoff procedure?
WGU uses the Modified Angoff procedure for setting cut (passing) scores for objective assessments, which provides a fair and legally defensible approach to setting scores. The Angoff procedure is a method for determining the difficulty of test items followed by a process for computing the exact passing score that is fair for each assessment.
WGU gathers a group of subject matter experts who know both the content of the assessment and a little about our students. These experts judge the difficulty of the test items based on how a group of 100 candidates (students) might perform. WGU then computes the average difficulty rating from the group for each of the test items. Finally, those item difficulty values are used to compute a cut score for each assessment. For example, if an assessment contains several difficult questions, the cut score might be set around 50%–60%. Conversely, if an assessment contains easier items, the cut score might be set above 80%.
What does this mean?
The cut (passing) score for each assessment is fair. It takes into account the difficulty of the test items and only requires students to achieve competency
Each version of an assessment (called forms) will contain different test items. A cut score is computed each time a new form is created to ensure fairness across versions of the assessments. WGU tries to balance the difficulty of items on each assessment form.
An assessment with a cut score of 55% is not any easier or harder to pass than another exam with a cut score of 85%. Both exams require students to achieve a score considered by experts to be at the level of competence.
RE: What does that cut score mean? - XanderMom - 05-06-2012 04:50 AM
(05-02-2012 05:27 AM)Dana Wrote: Thank you Mike, for the info and Mary for the suggestion of making it a sticky:
Thank you, Dana. I was wondering why the cut scores are usually pretty low. Wondering, but didn't research. Now it makes sense. You are so helpful! lol
Thank you again
RE: What does that cut score mean? - groove1 - 04-05-2014 05:49 AM
The argument against it is not the methodology; it is that is that competency level truly competent? For example, the first assessment - let's just call it a test; that's what it is regardless of semantics - I took at WGU had a cut score of 51%. I got 86% of the questions right on the pre-test with only reading about two of the seven chapters. I knew a little about the subject material but not a lot, but the two chapters I read made up enough of the test material for me to get 86% of the questions correct. Industry experts that teach these types of classes, such as Franklin Covey, would laugh at acing two of these chapters and that being considered "competent" in Organizational Behavior and Leadership. You don't even have to know about Leadership or Organizational Culture and you can still be considered competent in Organizational Behavior and Leadership. I wouldn't consider that "competent" in the subject material.
Ironically, after reading all of the material, I only got 65% of the questions right on the real test, but some of that was from environmental factors - such as my test being interrupted after I started it.
RE: What does that cut score mean? - John Tucker - 04-05-2014 06:47 AM
A cut score shouldn't be viewed similar to a percentage score. It's quite honestly difficult-to-impossible to determine how a grading team comes up with a cut score, as it's usually taken from a variety of different variables such as question difficulty, previous testing experiences, content specification and importance, etc.
WGU's objective exams are graded similar to how most professional certifications are, which is through a cut score. With the CPA Exam, we need a "75" on each of the four sections to pass and individuals who fail with a "60" or "70" constantly talk about how they just needed "5 or 10" more points to pass and it annoys me that these individuals don't understand the structure of the exam lol. If you didn't pass with the cut score, it's not that you missed by "1" point, it's that as an aggregate of the criterion used to confirm your competency, you didn't confirm "enough" competency. When you get a cut score of 75 and pass that section of the CPA Exam, that doesn't mean you got a C+ (75%) on the exam itself.
But yes, you could potentially pass a lot of WGU's Objective exams without studying that much if you pretty much already know the content material. Another way is if the content area is something basic, like Leadership, pretty much common sense can be used to answer most of those questions as anyone that's been in the workforce for more than 2 years should pretty much have a good understanding on basic Leadership concepts.
Do I agree with the cut score over the traditional grading system? Well, not really. The issue with the cut score is that you really have no idea what you got wrong per say, even though a report is shown with the general broad content areas that you were "weak" in, they really do very little to determine what you specifically missed and got correct. WGU could go into a point grading system, however, the drawback to this is that because you would know what you got wrong and right, you might not be able to just "take the test again" that easily with WGU. This might have WGU needing to add some sort of credit hour system.
RE: What does that cut score mean? - ratbuddy - 04-05-2014 09:00 AM
Er, the cut score is indeed the percentage of questions you need to get right to 'pass' the test.
RE: What does that cut score mean? - groove1 - 04-05-2014 09:34 AM
(04-05-2014 09:00 AM)ratbuddy Wrote: Er, the cut score is indeed the percentage of questions you need to get right to 'pass' the test.
Thank you. And at 51%, monkeys throwing darts at the answers could consistently score higher. WGU would consider them competent. You really could guess at the answers and have a darn good shot at being considered competent, even if you did not know the first thing about the material. That is the real issue.
RE: What does that cut score mean? - John Tucker - 04-07-2014 03:01 PM
The issue here though, is that if you view the cut score in the same context of a traditional grading system, there's no way 51% out of 100 would be the equivalent of a "B" (80% out of 100), in which WGU sets as the equivalent benchmark in passing individuals. If WGU were passing individuals scoring an "E" on a test by passing that off as a "B", then WGU would lose it's regional accreditation in a heartbeat.
It's why you have to keep in mind that the competency model is not the same as the traditional grading system. The traditional grading system wants to confirm you can memorize and regurgitate the content, the competency based grading system wants to confirm you can actually directly apply the content in the workplace. In both systems, you're graded much differently. It's why I compared it to the CPA Exam because it's pretty similar. One of the reasons we have to take BEC is that CPAs should know how to directly apply the accounting rules and principles to any business work environment. On top of that, it's why each section not only includes multiple choice questions but actual simulations that require problems to be worked out that demonstrate competency of the content. Even with the multiple choice questions, they get harder as you get more correct, which means getting 60% of the hard questions correct carry more weight than getting 100% of the easy questions correct.
With the competency model you are grading how well the content can be applied and if the individual confirms they can "most likely" apply the content in the workplace "above average" then they pass. WGU rounds up experts in the subject matter to determine which questions to put on the Objective exams including their level of difficulty. They creates rules such as if a person scores above "this minimal cut-off," then we will consider them "competent" enough to where if they stepped into the workforce tomorrow to perform these duties they should be performing above average work.
RE: What does that cut score mean? - ratbuddy - 04-07-2014 03:33 PM
Sure, it's on a curve decided by the content experts, but individual score still ranges from 0-100% depending how you do on the exam. It's not like the maximum score is any less than 'perfect.'
RE: What does that cut score mean? - groove1 - Today 10:59 AM
Ratbuddy is right. If you got every question correct, it would still be 100% in the traditional percentage or "for every hundred" context. Another issue I have already learned and what WGU actually teaches themselves is that people work much harder and learn more when a grade is at stake. With WGU's setup, you will do enough to pass, but not necessarily enough to earn a traditional A. This was taught in the first class I took at WGU yet they do not practice it.
Think about this. You go on a job interview and are up against somebody with similar qualifications, personality, and work ethic. The deciding factor is education. The other person has a resume with a 4.0 GPA from another accredited university. You have a degree from WGU which only shows that you are "competent". Who do you really think is going to get the job? Competency means enough to do it, but not necessarily enough to do it well.
RE: What does that cut score mean? - King Aires - Today 02:05 PM
(Today 10:59 AM)groove1 Wrote: Ratbuddy is right. If you got every question correct, it would still be 100% in the traditional percentage or "for every hundred" context. Another issue I have already learned and what WGU actually teaches themselves is that people work much harder and learn more when a grade is at stake. With WGU's setup, you will do enough to pass, but not necessarily enough to earn a traditional A. This was taught in the first class I took at WGU yet they do not practice it.
And you can get a degree from Harvard with a 3.0 by getting 2.0 in all your hard classes and 4.0 in electives. What is your point?
Again, if the experts out there are taking the test and they are only getting 57 on the test, then how do you expect a school to make their students get a 80 on that same test?
So basically you would prefer they increase the cut scores but throw in a bunch of 5th grade softball questions which does no one any good.
RE: What does that cut score mean? - ratbuddy - Today 02:13 PM
(Today 10:59 AM)groove1 Wrote: Think about this. You go on a job interview and are up against somebody with similar qualifications, personality, and work ethic. The deciding factor is education.
That's an edge case. If my degree from WGU gets me an interview, great, that's all I can ask for. It's up to me to talk my way into the job.
If my identical clone who went to MIT is there, screw it, he'll probably do a better job (and command a higher salary) than me anyway.