Abstract:
Software that easily helps higher education instructors to remove poor quality items and set appropriate grade boundaries is
generally lacking. To address these challenges, the SmartStandardSet system provides a graphical-user interface for removing
defective items, weighting student scores using a two-parameter model IRT score analysis, and a mechanism for standard-setting.
Stakeholders (i.e., teachers and students) from multiple faculties viewed and evaluated the system in a series of interviews and
focus groups. Generally, both groups of participants reported high levels of feasibility, accuracy, and utility in SmartStandardSet’s
statistical scoring of items and score calculation for test-takers. Teachers indicated the data displays would help them improve
future test items; students indicated the system would be fairer and would motivate greater effort on more difficult test items.
However, both groups had concerns about implementing the system without institutional policy endorsement. Students specifically
were concerned that academics may set grade boundaries on arbitrary and invalid grounds. Our results provide useful insights
into the perceived benefits of using the tool for standard setting, and suggest concrete next steps for gaining wider acceptance
that will be the focus of future work.
