The role of ethics review boards (aka Institutional Review Boards or IRBs) has long been discussed when considering social science, human factors, or usability studies. How much review is appropriate when the behaviours involved are limited to things people do in everyday life (e.g., trying a new computer program or completing a questionnaire)? Is the level of review done for medical experiments appropriate when it comes to usability research? This debate has flared up again with some recent rule changes in the US.
If you took Psychology 101 in college, you probably had to enroll in an experiment to fulfill a course requirement or to get extra credit. Students are the usual subjects in social science research — made to play games, fill out questionnaires, look at pictures and otherwise provide data points for their professors’ investigations into human behavior, cognition and perception.But who gets to decide whether the experimental protocol — what subjects are asked to do and disclose — is appropriate and ethical?
The dreaded memo has come around again – management has called another brainstorming session. They are getting people together to solve a problem or, more commonly, to discuss future trends or challenges for the organization.
You cringe when remembering the previous awkward sessions, when too many flip charts were filled with half-baked “ideas.” But the managers seemed to be pleased that “people were involved.” If you are an introvert, you remember the sessions as downright painful. And those flip charts that were so important that day – they were soon forgotten as people returned to their usual work.
But everyone says they had a fun day, so we know the next memo will come around soon enough.
Brainstorming has been described as the placebo in the management medicine kit – everyone believes that it works despite clear evidence that it does not. A recent Fast Company article has declared, “brainstorming is dumb.” Brainstorming continues to be popular even though major studies have shown that positive results are rare, at best, and viable alternatives are readily available.
Getting people together to propose new ideas seems, on the surface, like a good thing. However, as the Fast Company article describes, “just because you throw people together doesn’t mean wonderful things happen.” Others have been more critical, describing brainstorming as nothing more than executive entertainment.
So, what does the research say, and what are the alternatives to brainstorming?
To dig into this, we first have to review where brainstorming came from, and how it was supposed to work. Alex Osborn, an advertising executive, made brainstorming popular — most notably in his 1953 book Applied Imagination. Brainstorming was described as a wonderful group creativity technique that could be used to generate great ideas or solve hard problems. The term came from the idea of storming a problem like a group of commandos, with each “stormer” attacking the same objective.
Osborn was quite detailed in describing how brainstorming should be done. He established instructions for brainstorming sessions, many of which are now ignored. Osborn said that brainstorming sessions should:
Involve groups of about 12 people, or perhaps fewer
Avoid judging ideas
Strive for idea quantity, because quantity leads to quality
Promote divergent thinking and wild ideas
Welcome combinations and improvements on ideas
Perhaps the most overlooked principles in Osborn’s technique were:
Carefully choose the topic to be amenable to idea generation and as specific as possible
Provide pre-session training on the technique
Provide background information and orientation to the problem before the session
Osborn claimed that brainstorming, conducted with these principles, produced 44% more ideas than people working alone. With this evidence, and the attractiveness of the sessions, brainstorming took off and was widely adopted in many organizations.
Almost immediately, there were criticisms. Soon after Osborn’s book was published, researchers reported that, contrary to the doctrine, individuals often performed better than groups. Promoters of the technique argued, however, that those studies used questionable designs — one widely quoted study compared people working in groups versus individuals working alone, but all participants received brainstorming instructions. So, they argued, this was not a fair test of the technique.
Questions about how to properly test brainstorming remain today. Should we compare?
Brainstorming groups vs. brainstorming individuals
Brainstorming groups vs. other groups
Number of ideas vs. quality of ideas
Variety of ideas vs. novelty of ideas
With facilitator vs. without facilitator
Pre-existing groups vs. random groups
With so many open questions, it should be no surprise that studies that test brainstorming have come up with inconsistent results. But very often, the results are negative, suggesting that brainstorming may not actually work as advertised.
A review by Scott Isaksen examined 90 different studies and found that brainstorming was often not effective. The review also showed that many of the principles proposed by Osborn were not followed in the studies, leading to questions about the validity of the tests. Other studies have often found that individuals produced more, better ideas. Researchers have even proposed that working in groups could actually inhibit the creative process, perhaps due to nervousness, free riding by some participants, and “group think” where teams fixate on a few early ideas.
With the tests of brainstorming showing mixed results, people have started to consider modifications to the original Osborn principles. Some have argued that the lack of criticism, something that Osborn thought was crucial for allowing creativity to flourish, might actually be a problem. Their studies showed that adding debate and criticism to the brainstorming sessions actually led to more ideas.
Other people have started to question the role of the group interaction. With recent tech developments, researchers have started to examine electronic brainstorming, where people work at computers or online. When online, people can work anonymously while still participating in groups, perhaps using a shared editing space or chat service. Working electronically might reduce some of the inhibiting social factors of group work, such as nervousness or introverted personalities.
Another derivation of the original brainstorming technique is brainwriting where, instead of sharing ideas out loud in a group session, people write their ideas down and pass them around. Others can read the ideas and perhaps build on them, while continuing to work on their own ideas. The Fast Company article suggests that brainwriting, done best when writing both alone and in groups, can be a lot less dumb.
So, when the memo comes around again, propose an alternative? Try brainwriting or electronic brainstorming. Groups should exchange ideas when working on a problem, but they should also work alone, probably before they work in groups. The 1950s-style, awkwardly social, too-man-flip-chart sessions are not the best way to come up with good ideas.
Here is an interesting collection of tips for how to write well. Some are obvious, and some are rather novel and interesting.
Remember that the model of the drama is the dirty joke. This joke begins: “A traveling salesman stops at a farmer’s door” — it does not begin: “Who would think that the two most disparate occupations of agriculture and salesmanship would one day be indissolubly united in our oral literature? Agriculture, that most solitary of pursuits, engendering the qualities of self-reliance and reflection, and salesmanship, in which…”
SOUPS 2009 is underway in lovely Southern California. Google is hosting the conference this year.
SOUPS is the major conference in the field defined by the intersection of usability and security. The conference includes two tutorials, which took place yesterday, 15 technical research papers, a panel, break-out discussion sessions, and a keynote address.
I hate waiting in lines. If I go to a restaurant that has anything more than 5 people waiting, I leave. The thought of queuing up outside a store for one of those big holiday sales is completely foreign to me. I often arrange my daily schedule to avoid waiting in lines: I drive in off-peak hours, eat when others are working, and avoid crowded shopping centers in favor of small locations.
The one thing more frustrating that waiting in a line is waiting in a bad line. You know, those lines that are poorly designed, without clear guidance about where to wait, or what you are waiting for. Or lines where it is clear that there are not enough services at the end (cashiers, clerks, or kiosks) to handle the number of people waiting.
What makes this so frustrating is that waiting lines don’t have to be so terrible. There is a science of waiting lines, and if organizations follow the science they can make their lines more efficient and more pleasant. Waiting times are a necessary evil. Having some form of a queue is often the best way to ensure that a system or service works as efficiently as it can. Queues make sure that each resource, be it a cashier or a customs official, never goes idle waiting for the next customer. Queues that are too long can drive customers away, but queues of the right length can increase efficiency and save money.
Donald Norman, a popular usability guru, has recently outlined some valuable design principles for making good waiting lines. Many of these are common sense, but when they are not followed the effects are usually obvious, and unpleasant.
One of the most important principles is that human emotions dominate the experience of waiting in lines, and that emotions are affected by context. Setting a pleasant context for a wait in a line can make all the difference. A location that is warm, bright, cheerful, and welcoming will be far better tolerated. This Disney theme parks, which are synonymous with long lines, are very good at setting a good context for a waiting line, often having costumed characters entertain people while they wait.
My son, who has more patience than me, recently got into one of those multi-hour waiting lines for a Boxing Day sale at an electronics store. The store did many things to make the wait as pleasant as possible: they gave out coffee and snacks, they gave people a number so they could take a break and return to the line, and they held raffles for free merchandise. This is also a good example of a related design principle: keep people occupied. Peoples’ perceptions of time and space are influenced by their surroundings. Many service environments, such as waiting rooms, now have TV’s and advertising displays to keep people occupied.
Emotions spread from person to person. So, in managing a good waiting line, it is important to watch for and address any negative emotions. This means paying attention to the most upset customers so that their negative emotions do not spread to the other people waiting in the line.
Another design principle is to eliminate confusion by making it clear and unambiguous where the line starts, where it ends, and how it works. Perhaps the worst waiting experience is finding out you have been waiting in the wrong line. Some of the best lines I have seen at airports are where one employee at the head of lines provides reassurance that people are entering the correct queue.
Providing feedback on how the line is progressing is also important. Nira Munichor and Anat Rafaeli from Isreal recently conducted a study of telephone waiting lines. They had participants either listen to music while they waited for a telephone service, or the people heard music and one of two spoken messages. The first message apologized for the wait and asked people to remain on the line. The second message provided information about where the person was in the line (e.g., “You are currently third in line”). Call abandonment rates (the frequency by which callers hang up) were 50% lower with the informative position information than with the music or apology conditions, and the customer experience was described as much better. It is possible to design good waiting lines, even for the telephone.
Waiting lines also need to be appropriate: long waits for trivial reasons or trivial goals won’t make any sense. Waits also have to seem fair. If there are a lot of people waiting and yet only a few service locations open, such as at a bank, then this appears to be unfair.
A lot can also be done to improve the waiting line experience by changing the way the line works. Certain kinds of layouts can make lines more efficient, faster, and therefor more pleasant. You may have seen cashier stations at cafeterias, for example, that have two sides for serving customers on the left and the right. This is because the process of cashing out a customer can be inefficient, with time spent while the customer puts items on the counter, finds money, packs up the items again, etc. While a customer is busy doing those things on one side, the cashier can serve another customer on the other side. For situations like this, this kind of arrangement makes the lines work much better.
Another example of changing the way a line works can be seen in drive-through restaurant windows. These restaurants often have one place for giving a food order, another place for paying for the food, and a third place for picking up the food. And, these windows are often far apart. This physical distance means that, as long as there are a few customers, there are built-in delays in the process. This is important because preparing the food takes time, but by designing the line in this way customers are seldom left waiting for their food to be prepared — by the time they have visited all the windows the food is ready.
Another design decision is having multiple servers with multiple lines, as is usually done in super markets, or multiple servers and one line, as is usually done in banks. One line can appear to move faster and appear more fair, and a single waiting line automatically adapts if one of the servers is slow or encounters an unusual circumstance (one of those people with dozens of coupons).
Of course, sometimes the best design is to avoid the line in the first place. Reservations are a method of avoiding waiting lines. I recently traveled to Europe and visited a number of museums and art galleries. I did a lot of research to make as many reservations as I could, even if it cost more, to avoid the lines. Even though every guide book I read explained how to do this, I was amazed at the thousands of people who ignored the advice and waited in line after line. If you get to Rome and want the visit the Vatican museum, make a reservation for one of their wonderful guided tours and don’t wait in the infamous lines (and don’t miss the Borghese Gallery, where you have to have a reservation).
An interesting article from Wired on engineering versus design at Google. Douglas Bowman, Google’s visual design lead, is leaving because he says there is too much emphasis on engineering and not enough on design. While I am a big fan of having empirical evidence when making decisions, I can understand where it could be frustrating to have to run studies to decide on the width of a line or a shade of blue.
… he got fed up with constant pushback from a bureaucracy that seeks empirical justifications for choices made within what is essentially is an art form — and often about relatively insignificant details.
Recently, the FTC and eight federal regulators of the financial industry in the US have proposed adoption of a model privacy notice form. This form would be used by financial institutions to inform customers about the institution’s privacy practices, and provide opt-out opportunities for the sharing of personal information.
This model privacy notice was developed using iterative, user-centred research and development. A report by the Kleimann Communication Group describes the research that went into the prototype. The goal of the project was to develop a paper-based privacy notice that was comprehensive, comprehensible, standardized to allow comparisons, and compliant with existing regulations.
The research and development process was conducted over 16 months and included 2 focus groups of 10 people each, preference testing with 7 participants, pre-testing with 4 participants, and diagnostic usability testing with 35 participants in 5 US cities. The model notices were revised during each of these steps. Page 1 of the final 3 page form is shown below.
The prototype privacy notice contains 4 main sections: (1) a “key frame” that answers generic Why, What, and How questions concerning the sharing of personal information; (2) a disclosure table that states the practices of the specific financial institution using the form (e.g., information is shared with affiliates for marketing purposes), and whether the customer can control those practices (i.e., opt-out options); (3) a secondary frame that provides definitions (e.g., “affiliates”) and answers to frequent questions (e.g., Why can’t I limit sharing?); (4) and an opt-out form where customers indicate their privacy choices.
The final prototype notice appears to be a usable and flexible tool for gaining understanding and consent. Follow-up evaluation is being planned once the notices have been used with the general public.
The development methodology and the resulting model forms might be applied to other areas where notice and consent are required. For example, participant consent forms used during research on human subjects are often overly long and complicated, often with the intent of appeasing an ethics review board rather than informing the participants. Perhaps this approach used for privacy notices could be used to improve and standardize these consent forms.
If you are a home-owner struggling to understand why your electricity bills are so high then you may be interested in a new product from Blue Line innovations called the Power Cost Monitor.
The company tells Digital Home that tests in Ontario and elsewhere have shown that homeowners who use the real-time feedback device often reduce their electricity use by as much as five to 20 per cent.
The PowerCost Monitor is a real-time direct feedback display device that informs consumers how much electricity is currently being consumed in their home along with the total electricity used in dollars and cents and in kilowatt hours.