Update 2: 25 January @ 10:30 pm Pacific
More research, new lede: From 2013 to 2014, there was a slight decrease in the percentage of Americans who are “Bible-minded.”
If I told you that a survey suggested that the deep south was more religious-minded than the northeast or the far west, I don’t think you’d be surprised.
In a nutshell, that’s the claim made by the latest “Bible-minded” study from the American Bible Society. The report also noted an inverse relationship between population density and being “Bible-minded.”
But as I’ll show, there are serious questions about the methodology.
What’s does Bible-minded mean, you ask?
Respondents who report reading the bible within the past seven days and who agree strongly in the accuracy of the Bible are classified as “Bible Minded.” (emphasis added)
Here we go … drum roll, please.
Most Bible-minded cities (alleged)
- Chattanooga, TN
- Birmingham, AL
- Roanoke/Lynchburg, VA
- Springfield, MO
- Shreveport, LA
- Charlotte, NC
- Greenville/Spartanburg, SC / Asheville NC
- Little Rock, AR
- Jackson, MS
- Knoxville, TN
Least Bible-minded cities (alleged)
- Portland/Auburn, ME
- Burlington/Plattsburgh, VT
- Phoenix/Prescott, AZ
- Hartford/New Haven, CT
- Buffalo, NY
- Cedar Rapids/Waterloo, IA
- San Francisco/Oakland/San Jose, CA
- Boston, ME / Manchester, NH
- Albany / Schenectady / Troy, NY
- Providence, RI / New Bedford, MA
I confess that my statistics classes in grad school did not prepare me for this.
First, these “city” groupings are inexplicable.
Some are cities. Some are Metropolitan Statistical Areas. Others are just weird.
For example, Providence doesn’t belong with New Bedford — they are 30 miles apart and in two different MSAs. Roanoke and Lynchburg are 60 (long) miles apart. The Phoenix MSA does not include Prescott, which is 100 miles to the north!
Portland and Auburn are almost 40 miles apart — and we’re talking Maine, here. Rural.
And do they really mean
there is no Manchester, NH? (There’s a closer one in MA and one in VT, too.) [See comments re correction.]
Maybe this is supposed to represent “a tie”? That means, instead of 100 cities, we have 168 cities?!?
The data reported is based upon telephone and online interviews with nationwide random samples of 46,274 adults conducted over a seven-year period, ending in August 2013.
I interpret the statement of methodology like this: researchers interviewed about 6,500 adults last year. And that 46K number over a 7-year period? Time-series data. Doesn’t tell squat about the status quo in 2013. In the aggregate, it could possibly tell us something about national trends.
There are 100 “cities” in the report. Dividing by 100 (not a recommended way to architect a meaningful survey) yields only 65 people per city per year.
Even if the “cities” has similar populations, this sample size is not sufficient to generate a ranking that has any meaning. And since we seem to have 168 cities, meaning falls further into a dark abyss.
We know these cities aren’t the same size, but the press folks want us to be certain:
Of the top 25 Bible-minded markets, only three have a population of greater than 1 million households: Charlotte, N.C.; Nashville, Tenn.; and Dallas.
Oh. Now they are markets, not cities. But markets not based on MSAs. Back to an odd 100 points of division.
Maybe the researchers did interview almost 50,000 people each year. I don’t think so, but that’s still only 500 people per city (were they divided equally). Still not enough for a random sample to be very meaningful. Random samples of this size are hardly worth the cost of the electricity needed for you to read this article if the researchers slice-and-dice them into 100 groups!
Moreover, views on the “accuracy” of the Bible — using evolution as a proxy — are known to vary significantly by both religious group and political party affiliation. Once again, random sampling doesn’t reveal much, statistically. This is where you need multivariate analysis, in my opinion.
About the margin of error
Note that the margin of error statement associated with the data is nuanced:
The maximum margin of sampling error associated with the aggregate sample is ±0.5 percentage points at the 95% confidence level (emphasis added).
The aggregate sample is all the data, not the data sliced-and-diced into 100 “cities” and then divided by “who reads the bible regularly and believes it literally.” It’s like national political polling data, where the margin of error grows larger when extrapolating to a specific state.
Specifically, a number (not reported) of x% of Americans being “Bible-minded” would have a small margin of error if the data were collected in a reasonably constrained period of time, which is usually a period of days, not weeks. This is analogous to a pre-election report (usually representative, not random) that N% of Americans supported Obama versus M% behind Romney.
But state-by-state analysis? There the margin of error grows larger. And this study doubles the number of increments.
Finally, a random sample of 46K adults taken in a specific (short) period of time should have a very very small margin of error when looking at specific characteristics of the entire group.
But that’s not this study.
Why is this a problem?
Because mainstream (read that as having a megaphone) media picked up this story! An era of 24×7 news coverage, compounded by the need to tweet and Facebook and-and-and more than once an hour, means that fluff gets picked up.
- NPR rewrote the news release and didn’t look at the chart for city names/groupings.
- The TIME headline screamed “These are the most godless cities in Amercia” (the society took great pains to distinguish Bible behavior from generalizations like this one). In addition to rewriting the news release, the blogger embedded the Society’s infographic. And the lede: “America, you may have a new Sodom and Gomorrah.” (Downworthy)
- CBS Chattanooga: Chattanooga Named Most ‘Bible-Minded’ City, citing as its source Christianity Today (kinda makes the point of the survey, doesn’t it?)
How can we expect normal people to think critically about tweets and Pinterest pictures and Facebook memes if reporters don’t engage brain before fingers when reading News Releases?!?
Yes, there is one!
For example, 6-in-10 of us say we pray daily.
Evangelical churches have a slightly greater percentage of non-high school grads (16%) than the national population (14%).
Buddhists (26%), Jews (35%) and Hindus (48%) are much more likely to hold a graduate degree than the national population (11%).
And any possible real differences in Bible-study behavior can probably be explained by the geography of religious tradition. The Northeast is home to 41% of America’s Jewish faith and 29% of its Catholics but only 19% of the U.S. population. The South: 60% of the historically black churches and 50% of evangelicals; it’s only 36% of the nation’s population. The West has 76% of the Mormons and 29% of the unaffiliated, but accounts for only 22% of the nation’s population. And the Midwest is 23% of the population of the country and 29% of the mainline churches.
Enough about religion in America.
The moral of this story is simple: if something sounds too good or too bad to be true, then it probably isn’t.
Friends don’t let friends share suspect news and photos, no matter how good (or bad) it makes everyone feel.
I’ve found a reference to the 2013 study report, the first one. Much more, umm, precise language in some places.
- 96 (not 100) U.S. localities, which the consultant-generated infographic defines as Nielsen “designated market areas” — in other words, LOCAL TV MEDIA MARKETS. Not cities, towns or MSAs.
- Last year there were 42,855 interviews. This year it’s 46,274, which means my estimated 6,500 year-to-year growth was overly-optimistic. The total is a rolling number; that is, last year’s report also covered a 7-year period.
- Last year, the top 10 “cities” ranged from 47-52% of the population being “Bible-minded.” This year, the range was 44-51%. I don’t believe this difference is statistically significant, but the 0.5% claim could certainly be interpreted as “yes it is.”
- Last year, and this year, the bottom 10 “cities” percent of population considered “Bible-minded” ranged from 9-18%.
PS. Can someone please help me understand why this things is making my blood pressure hit the stratosphere?
Featured Photo: Creative Commons license
Known for gnawing at complex questions like a terrier with a bone. Digital evangelist, writer, teacher. Transplanted Southerner; teach newbies to ride motorcycles! @kegill, wiredpen.com