Collective intelligence & the ethics of A/B tests

Rationally Speaking | Official Podcast of New York City Skeptics - Current Episodes - RS 209 - Christopher Chabris on "Collective intelligence & the ethics of A/B tests"

"We have the concept of intelligence, the psychological concept of intelligence, as a measurable thing about people, because when you give a bunch of people a bunch of different cognitive tasks, it just turns out empirically that for whatever reason, people who do well on one of the tasks also tend to do well on the other tasks. They're not perfectly correlated, so it's not as though the person who gets the highest score on task one necessarily gets the highest score on all the other tasks and so on, but there's a general tendency for the performance on different kinds of cognitive tests to be positively correlated. We call the capacity, the inferred capacity that can lead people to do well on a variety of tasks, we call that intelligence...

There's a guy at Harvard Business School whose name escapes me right now, because I don't have his book in front of me. He did a study of equity research analysts on Wall Street in the '90s, but these are the people who analyze companies. They say buy, hold, or sell, and set price targets and things like that.Their performance is measured in — it's a little bit of a fuzzy metric — by how highly their customers rate them. Still, it turned out that when these analysts got very highly rated, they tended to be poached by competing banks who would hire them. When that happened, their performance tended to go down — but if they were women, their performance recovered faster than if they were men.One interpretation of that is that what you're really measuring here is not the performance of this one person, but the entire team that they're a part of. Therefore, there could be some effect of women adding more to team collective intelligence or something, that leads to better output...

I want to raise a general concern that I have about studies that find that women are better at something than men. My general concern is I worry that the opposite result would be unlikely to be published. If a researcher did a study that seemed to show, "Oh, hey, when you add women to a team, the team does worse," is that paper going to get published? It just seems so inflammatory that my suspicion is that either a journal would be reluctant to publish it, or would subject it to much more stringent standards to make sure it's a real result, to avoid publishing a false inflammatory study. Which — stringent standards are good, but if you're applying them unequally, then that affects the ratio of findings that you end up seeing. Or maybe the researcher himself or herself wouldn't try to pursue that finding because of the potential fallout...

'Companies being reluctant to run beneficial experiments from which they could learn a lot, because they don't want people to find out that they've been running an experiment. Instead, they don't run an experiment; they just go with lesser quality data or no data at all, or just intuition. Or as someone said, the “HIPPO,” the highest paid person's opinion, just governs the outcome.

That's to us, and probably to a lot of people, not the most enlightened way to figure out what policies and treatments and practices are likely to work best, either for the company, for the company's bottom line, or for their customers. Companies very often are concerned about the welfare of the customers. They honestly try to make products that improve people's lives, so everybody has a stake in this, in this illusion, I think.'...

'This reminds me of an old anecdote — I have no idea if this is real or not — where some prestigious, esteemed doctor ... I think it was a surgeon ... was reporting some change in surgical methodology and arguing that this would be a good thing, and some student in the back raised his hand and was like, "Why don't you try it on only half your patients to see if it works well?"

The presenting surgeon took umbrage at this, and he was like, "You're seriously telling me that we should subject half of our patients to worse treatments just for the sake of experimentation? I'm not going to subject half the people to worse treatment."And the student just replied, "Which half?"'


On the 'myth' of IQ