Commentary on new usable security research: The Emperor is biased

I have recently posted a long commentary on some new research by Stuart Schechter, Rachna Dhamija, Andy Ozment, and Ian Fischer. Their paper, titled The Emperor’s New Security Indicators: An evaluation of website authentication and the effect of role playing on usability studies, has been receiving a lot of attention because it shows that people ignore new security indicators being used by banks to prevent phishing attacks.

emperordata
I think there are serious problems with the methodology in this research caused by a failure to understand the psychology of research participation. As a result, I think the results are biased in the direction of providing over-estimates of the real-world rates at which these security indicators will be ignored. My motivation in this commentary is to discuss the issues associated with this kind of research methodology; so that we can all do better research.

There are at least 3 well-know psychological phenomena related to participation in research studies that are important here: demand characteristics, task focus, and obedience to authority. In this essay I review these phenomena, explain how each one could have biased the results, and provide concrete suggestions for improving this, and similar, research.

Please have a look at the essay at http://www.andrewpatrick.ca/essays/commentary-on-research-on-new-security-indicators/
and provide comments there.