On trying to dismiss research findings you don't like (OR: On women being more religious than men)

"I never put on a pair of shoes until I've worn them at least five years." - Samuel Goldwyn

***

Comments on a post on Token Skeptic about Women And Paranormal Beliefs:

Me: I think any investigation of gender differences in skepticism cannot ignore the fact that women are more religious than men.

DSKS: Women are not more religious than men. Much like the intelligence difference issue, there’s a palpable irony in the fact that the statistics of such studies are so often approached by men with exactly the kind of lazily intuitive thinking that the data ostensibly suggest women are the more prone to.

As far as I can tell there appears to be a tenacious trend indicating that for every 7 in 10 fellas who are religious (depending on the strength of definition) maybe 8 in 10 women are. At best, that suggests there is a slightly higher likelihood that a randomly sampled woman will be “religious” than a randomly sampled man, which is a completely different conclusion than “women are more religious than men” (which is demonstrably false).

Nevertheless, a difference of 1 in 10, whether it passes muster according to a given statistical model or not (and notwithstanding the hell of controlling for all the biases in the study), is still a pretty lousy difference, and certainly not one upon which a physiological argument from gender can be clearly made.

Statistics 101: just because a difference is significant doesn’t necessarily mean that it’s significant.

Me: You need to provide evidence to support your claim that a statistically 14% difference is not significant

14% is very much bigger than what is found in many studies whose results are accepted as uncontroversial

Furthermore, the finding holds true across a wide variety of metrics:

http://www.livescience.com/7689-women-religious-men.html

“The percent of women (and then men) who:

* Are affiliated with a religion: 86 (79).
* Have absolutely certain belief in a God or universal spirit: 77 (65).
* Pray at least daily: 66 (49).
* Have absolutely certain belief in a personal God: 58 (45).”

This gender disparity also shows up across 7 decades of polls.

All this points to a remarkably robust result which requires, at the least, a great deal of explanation to challenge.

DSKS: Well, I was being facetious, but with regard to providing evidence for a particular arbitrary interpretation of statistical data versus another… is there evidence that appealing to p<0.05 provides for more accurate conclusions than p<0.01? Type I or Type II error, one must choose one's poison.

Of course, whether an absolute but relatively small difference that appears unlikely to be due to random variance in single population (P<0.whathaveyou) is, for want of a separate word, informative, is thoroughly context-dependent (and no where is this a more important issue to acknowledge than in the already dubious realm of evopsych).

Here’s some arguments for why the hypothesis that this small difference between men and women is due to some intrinsic biological predisposition towards religiosity should be taken with a pinch of salt. First, relative to the kinds of differences we usually attribute to the XY/hormonal differences underpinning the separation of man and woman, the differences highlighted above are not very compelling and even less informative. In the context of sex differences, which are usually profound (e.g. differences in endocrine function, skeletal structure, chromosome configuration &C) it can at least be said that for a given property, either most women have it and most men don’t or vice versa. In this instance, most men and women have it (religious leanings), but we still want to conclude that the slightly increased chance of a woman being religious is indicative of a biologically based gender distinction. That, in a room with 10 men and 10 women, we will find that there will be one more woman with religious tendencies than there are men. That might be informative if, of the 20 individuals, only a single woman had any religious leanings at all, although even here it would indicate that religiosity was merely an aberration that was sex specific, rather than a norm for that sex. But in a room in which 7/10 men and 8/10 women are religious… well, it’s not quite time to start digging deeply into the psychology of women for want of an explanation for their irrational leanings, imho. To think otherwise is to be guilty of the kind of shoddy inductive reasoning that has plagued psychology for centuries all the way up to the middle of the last one (and more often than not to the detriment of women and minorities incidentally).

Second, there is the highly important issue of adequately controlling for other factors that might equally contribute to these small differences. Crucially, there is no population of women that we can turn to to provide a control group of females that has existed in state of complete and total equality with their male counterparts, and thus by which we could reasonable remove the social baggage that invariably taints such investigations.

Third, but related to the second, religious institutions are almost all patriarchal, and almost all hang on the prophesies, testimonies, miracles and other arcane meanderings of… men. Which puts a big bloody question mark over the idea that women are, by pure voluntary association, more likely to be religious. The shadow of coercion, so neatly emphasized by such thrilling events as the Salem Witch trials, is strong and it is thoroughly male. That the religious leanings of that extra woman in the group of 20 are due to a profound social pressure does not seem like an outrageous hypothesis. Certainly a better one than “Women are more religious than men because they’re women”.

Me: A p-value of 5% is standard in social science and is a good compromise between Type I and Type II errors. That’s the reason why good research must be replicable – and 7 decades of polls is surely sufficient for that. In any event, this is a very simple research finding (very unlike studies of whether drugs are more effective than placebos) with an effect size larger than what you typically find in research, so I am sure the p-value is far below 1%.

As for biological differences, when did they come into the picture?! What I was referring to was the very uncontroversial finding that women are *more religious* than men, not that there’re inherent biological reasons why this is so. There are various theories put forward to explain why women are more religious than men, and not all of them are biologically grounded.

When you sample size is 20 – 10 in each population – I would not draw any strong conclusions. Yet when we crank up the sample size – in theory 30 is the minimum you need to get a reasonably accurate result – and more importantly, replicate the findings multiple times, attempts at denying findings you don’t like look more and more like delusion (the social baggage that invariably taints such investigations is not always on the part of those seeking to draw conclusions from research).

I’m assuming that you don’t think very highly of social science research in general, since more or less all of it can be objected to on similar grounds.

DSKS: I have no general aversion to social science. Arguably the best and the worst use of statistics occurs within that discipline. (Your last post suggests that you might be of a frequentist disposition, which is interesting because this approach is currently under pressure in the social sciences, assailed as it is by the Bayes brigade.

However, as I understand it the majority of these studies have not been conducted to the standards of a serious quantitative study by a team of social science researchers anyway; mostly basic Q&A based polls from Gallup and similar outfits. Outfits that have been notoriously wrong in there poll-based predictions for more concrete things like voting patterns. These also tend to return the kinds of numbers that are ripe for cooking in all sorts of ways to yield different strengths of interpretation. e.g. for the following:

* Are affiliated with a religion: 86 (79)
* Have absolutely certain belief in a God or universal spirit: 77 (65).
* Pray at least daily: 66 (49).
* Have absolutely certain belief in a personal God: 58 (45).”,

Assuming equal sampling of men vs women, the female fraction of religiously affiliated, God believing, and praying are 52, 55, and 57% respectively. Suddenly, the differences aren’t so striking, and when we consider that the latter three questions are a little vague (what exactly does an individual consider to be “God” or a “universal spirit” and what constitutes prayer?) they’re even less so simply by virtue of being difficult to parse meaningfully.

At best we can say that there is possibly a weak but persistent trend indicative of higher probability of a randomly selected woman being religious than for man. But simply stating a frequency statistic that, “More of the religious are women” is very different from the statement and conclusion that, “Women are more religious than men”. Given as a hypothesis, the latter is as immediately falsifiable as the statement, “Men are taller than women”.

Me: Again, your objections are not unique to Gallup and other pollsters. Yes they have been wrong in the past, but just because you are wrong sometimes does not mean that your results cannot be trusted.

Election polling is especially noticed when it is wrong. Which means that it is right often enough for us to place weight on it. Indeed, the record of election polling is generally good (and almost always right in the results of the election, even if not in the margins of victory: http://www.gallup.com/poll/9442/election-polls-accuracy-record-presidential-elections.aspx)

The widely accepted test size of 5%, after all, means that there is a 5% chance of getting the results in question even if your null hypothesis is true. I’m assuming your bar for “a serious quantitative study” is really high, and would disqualify a good deal of research; despite repeating the mantra of “correlation is not causation” when they don’t like particular research findings, I still see researchers fall prey to it – especially when they like what they find (one example: TV violence).

And just mentioning Bayesianism does not mean that women are somehow not more religious than men – after all, the statistics show that more women are religious than men, which will update your prior probability

While definitions of prayer and gods differ, we are not trying to investigate the research question, “Do more women than men hold to the Nicene Creed” but simply “Are women more religious than men?”. What sort of questions would you ask to determine religiosity, if not these?

Also, differences do not have to be overwhelming in order to make a difference. To turn it around, the gender wage gap in 2008 was 77:100 (this is ignoring very important factors like education, experience, occupation, industry and union membership). So the female fraction of earnings in 2008 was only 44%. Yet, if I said “Suddenly, the difference is not so striking” or that “at best we can say that there is possibly a weak but persistent trend indicative of higher probability of a randomly selected woman earning less than a man”, you can imagine the frosty reception I would receive.

Lastly no one (except those flogging straw men) seriously thinks that the statement “Men are taller than women” means that all men are taller than all women, or that “Women are more religious than men” means that all women are more religious than all men, any more than anyone would say that the claim that “there is racism against blacks in the United States” is falsified by there being a black President.