A quote from "The Trouble With Computers" about usability studies at Apple when they were developing the original Macintosh:

Apple interface guru Bruce Tognazzini tells this story. The in-box tutorial for novices, "Apple Presents... Apple," needed to know whether the machine it was on had a color monitor. He and his colleagues rejected the original design solution, "Are you using a color TV on the Apple?" because computer store customers might not know that they were using a monitor with the color turned off. So he tried putting up a color graphic and asking, "Is the picture above in color?" Twenty-five percent of test users didn't know; they thought maybe their color was turned off. Then he tried a graphic with color named in their color, GREEN, BLUE, ORANGE, MAGENTA, and asked, "Are the words above in color?" Users with black and white or color monitors got it right. But luckily the designers tried a green-screen monitor too. No user got it right; they all thought green was a fine color. Next he tried the same graphic but asked, "Are the words above in more than one color?" Half the green-screen users flunked, by missing the little word "in". Finally, "Do the words above appear in several different colors?" Success.

That was what Apple did for a single throwaway UX question in a non-core part of the product - before its first release. It apparently took about 5 iterations of UX design followed up by UX research before they finally converged on the right answer.

The lesson I learned from that: usability studies are important. But you can't just take the recommendations of a usability study; you have to implement the recommendations, do another study, be prepared to be frustrated that the new version is just as bad as the old one, and do it all again. And again. If that's not how you're doing usability studies, you're doing it wrong.

Maybe I should re-buy that book. I gave mine away at some point. It's kind of indispensable as a tool for explaining software usability research, if only for its infamous "Can you not see the cow?" photo.