Stark, Stephen; Chernyshenko, Sasha and Guenole, Nigel. 2011. Can subject matter experts' ratings of statement extremity be used to streamline the development of unidimensional pairwise preference scales? Organizational Research Methods, 14(2), pp. 256-278. ISSN 1094-4281 [Article]

No full text available

Abstract or Description

Interest in on-demand noncognitive assessment has flourished due to advances in computer technology and studies demonstrating noteworthy predictive validities for organizational outcomes. Computerized adaptive testing (CAT) based on the Zinnes-Griggs (ZG) ideal point item response theory (IRT) model may hold promise for organizational settings, because a large pool of items can be created from a modest number of stimuli, and the items have been shown to be resistant to some types of rater bias. However, sample sizes needed for marginal maximum likelihood (MML) estimation of statement parameters are quite large and could thus limit usefulness in practice. This article addresses that concern and its ramifications for CAT. Specifically, we conducted empirical and simulation studies to examine whether subject matter expert (SME) ratings of statement extremity (location) can be substituted for MML estimates to streamline test development and launch. Results showed that error in SME-based location estimates had little detrimental effect on score accuracy or validity, regardless of whether measures were constructed adaptively or nonadaptively. Implications for research involving small samples and CAT in field settings are discussed.

Item Type:	Article
Identification Number (DOI):	https://doi.org/10.1177/1094428109356712
Departments, Centres and Research Units:	Institute of Management Studies Psychology Research Office > REF2014
Dates:	Date Event 2011 Published
Item ID:	5295
Date Deposited:	16 Mar 2011 15:26
Last Modified:	27 Jun 2017 13:10
Peer Reviewed:	Yes, this version has been peer-reviewed.
URI:	https://research.gold.ac.uk/id/eprint/5295

Edit Record (login required)