Open Access Repository

The Quality of Response Time Data Inference: A Blinded, Collaborative Assessment of the Validity of Cognitive Models


Downloads per month over past year

Dutilh, G, Annis, J, Brown, SD, Cassey, P, Evans, NJ, Grasman, RPPP, Hawkins, GE, Heathcote, A ORCID: 0000-0003-4324-5537, Holmes, WR, Krypotos, A-M, Kuptiz, CN, Leite, FP, Lerche, V, Lin, YS ORCID: 0000-0002-2454-6601, Logan, GD, Palmeri, TJ, Starns, JJ, Trueblood, JS, Visser, I, Voss, A, White, CN, Wiecki, TV, Rieskamp, J and Donkin, C 2018 , 'The Quality of Response Time Data Inference: A Blinded, Collaborative Assessment of the Validity of Cognitive Models' , Psychological Review , doi: 10.3758/s13423-017-1417-2.

PDF (Author manuscript)
124374_The Qual...pdf | Download (301kB)

| Preview


Most data analyses rely on models. To complement statistical models, psychologistshave developed cognitive models, which translate observed variables into psychologicallyinteresting constructs. Response time models, in particular, assumethat response time and accuracy are the observed expression of latent variablesincluding 1) ease of processing, 2) response caution, 3) response bias, and 4) non–decision time. Inferences about these psychological factors, hinge upon the validityof the models’ parameters. Here, we use a blinded, collaborative approach to assessthe validity of such model-based inferences. Seventeen teams of researchersanalyzed the same 14 data sets. In each of these two–condition data sets, we manipulatedproperties of participants’ behavior in a two–alternative forced choice task.The contributing teams were blind to the manipulations, and had to infer what aspectof behavior was changed using their method of choice. The contributors choseto employ a variety of models, estimation methods, and inference procedures. Ourresults show that, although conclusions were similar across different methods, these“modeler’s degrees of freedom” did affect their inferences. Interestingly, many ofthe simpler approaches yielded as robust and accurate inferences as the more complexmethods. We recommend that, in general, cognitive models become a typicalanalysis tool for response time data. In particular, we argue that the simpler modelsand procedures are sufficient for standard experimental designs. We finish byoutlining situations in which more complicated models and methods may be necessary,and discuss potential pitfalls when interpreting the output from response timemodels.

Item Type: Article
Authors/Creators:Dutilh, G and Annis, J and Brown, SD and Cassey, P and Evans, NJ and Grasman, RPPP and Hawkins, GE and Heathcote, A and Holmes, WR and Krypotos, A-M and Kuptiz, CN and Leite, FP and Lerche, V and Lin, YS and Logan, GD and Palmeri, TJ and Starns, JJ and Trueblood, JS and Visser, I and Voss, A and White, CN and Wiecki, TV and Rieskamp, J and Donkin, C
Journal or Publication Title: Psychological Review
Publisher: American Psychological Association
ISSN: 0033-295X
DOI / ID Number: 10.3758/s13423-017-1417-2
Copyright Information:

© 2018 The Authors. The final published version is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0) author manuscript is © American Psychological Association, 2018 This paper is not the copy of record and may not exactly replicate the authoritative document published in the APA journal. Please do not copy or cite without author's permission. The final article is available, upon publication, at: 10.3758/s13423-017-1417-2

Related URLs:
Item Statistics: View statistics for this item

Actions (login required)

Item Control Page Item Control Page