The complete Next Mormons Survey questionnaire script can be downloaded here.

 

The explanation that follows is an abbreviated version of the appendix of The Next Mormons by Oxford University Press, authored by Benjamin Knoll and Jana Riess. Readers who are interested in more detail can find it in the back of the book. We are grateful to Oxford University Press for giving Jana permission to reprint some of the appendix material here.

Data Sources and Funding

The 2016 Next Mormons Survey (“NMS”) is an online public opinion survey designed and fielded by Jana Riess and Benjamin Knoll in 2016. We contracted with the survey firm Qualtrics, an online data collection firm based out of Seattle, Washington and Provo, Utah, to gather responses for this survey. Funding for this contract came from a Kickstarter campaign fielded by Jana Riess from July 15 through July 29, 2016. In all, the campaign raised $19,665 from 245 individual donors, which was supplemented by more than $6,000 in additional donations after the Kickstarter campaign had ended. (Click  here for a donor list.) The NMS is an independent research study that is not funded by or affiliated with the institution of the Church of Jesus Christ of Latter-day Saints.

We chose Qualtrics and its online data collection method for several reasons:

1. It is very difficult to reliably measure public opinion among small subsets of a population using traditional random-digit-dialing telephone survey methods. This is because when analysts want to measure public opinion of a particular group, it is necessary to obtain a randomized sample so that everyone in the population has an equal chance of participating in the survey. Mormons constitute less than 2 percent of the American population, making it extremely costly to gather a sufficiently large sample from such a small group using traditional telephone-based survey methods.
2. The NMS is lengthy, with more than 130 separate survey questions in all (though not all questions applied to every respondent). It is much faster to complete the survey online versus over the telephone.
3. Online data collection firms often reward their respondents for completing surveys and thus require that all questions are answered. Incentives for taking surveys can include points to earn store gift cards or even small cash payments. This results in fully completed surveys that do not suffer from the same “missing data” problem as many other telephone- or internet-based voluntary surveys (see here for more information).

These various issues make online survey firms such as Qualtrics a viable option for studying small populations that are difficult to capture in traditional telephone-based surveys. These firms use a “panel matching” technique to acquire sufficient responses. Surveyors can specify a variety of demographic or response quotas to increase the representativeness of the survey respondents to the population of interest. Research has shown that online samples from reputable firms such as Qualtrics produce samples that are comparable in representativeness to randomized telephone surveys. (Note that this approach differs slightly from the “sample matching” technique employed by YouGov and other firms. With sample matching, respondents are more precisely targeted based on an algorithm of desired sample characteristics. The panel matching method instead specifies certain quotas for desired sample characteristics and continues to field surveys until the quotas are reached.)

Sample Size and Margin of Error

The NMS was in the field from September 8 to November 1, 2016, though the majority of responses were collected during September. In all, 1,156 self-identified Mormons were included in the final sample, as well as 540 former Mormons, for a total of 1,696 completed surveys. The current Mormon sample has a standard survey margin of error of 2.9 percent and the former Mormon sample one of 4.2 percent, based on the sample sizes and the estimated size of those populations in the United States. For simplicity, we consider the margins of error to be ± 3 percent and ± 4 percent, respectively. The survey design and question wording received approval from Centre College’s Institutional Review Board (IRB) on September 1, 2016 (Centre College IRB Assurance #FWA00017871; IRB approval code 140-Knoll-NMS-F16).

How We Define Mormons and Former Mormons

The LDS Church defines its members and former members by their status on official membership registration rolls, regardless of their level of activity and social or emotional attachment to the faith. In contrast, public opinion researchers of social topics such as religion must rely on survey respondents to describe their own demographic characteristics and are not usually able to independently verify the accuracy of these self-reported responses. In our case, we allowed respondents to self-select into the survey based on the nature of their identification with Mormonism. This means that “current Mormons” and “former Mormons” in our survey are those who say they are Mormons (or once were) regardless of their current status on the LDS Church’s membership rolls. Using self-identification as a basis of survey categorization is a standard procedure for social science public opinion research data. To find these self-identified Mormons, all potential survey respondents received this initial screening question:

Do you currently identify, or have you ever identified, as a member of the Church of Jesus Christ of Latter-day Saints (Mormon)? Please read these options carefully before selecting.

1. I currently identify as LDS and was LDS for at least one year before turning 18. This could include:
   • Being born and raised in the LDS Church.
   • Converting to the LDS church before age 17.
2. I currently identify as LDS but joined the LDS Church only after turning 17.
3. I identified as LDS at one point in my life for at least one year starting before age 18 but no longer do. This could include:
   • Being born in an LDS family but no longer identifying as LDS.
   • Converting to the LDS church as a child/adolescent and being involved for at least a year but no longer identifying as LDS.
4. I identified as LDS at one point in my life for less than one year starting before age 18 but no longer do. This could include:
   • Being born in an LDS family but my family stopped involvement/identification with the LDS church while I was still a baby.
   • Converting to the LDS church as a child/adolescent and staying involved for a few weeks or months but ceasing involvement with the LDS Church within a year.
5. I currently identify as a member of the Community of Christ or Fundamentalist Church of Jesus Christ of Latter-day Saints.
6. None of the above.

The vast majority of the 23,080 potential respondents queried by Qualtrics selected “none of the above” and were thus excluded from the final survey sample. Those who indicated identification with the Community of Christ or FLDS Church were likewise excluded. While adherents of those faiths certainly qualify as members of the wider Mormon tradition, we were specifically interested in members of the Latter-day Saint branch for the purposes of this project. We also excluded from the survey those who said that they identified as LDS for less than a year as children or adolescents but no longer do so, as the sample of interest was specifically those who identified as LDS long enough for it to have a substantive impact on their religious and social lives. Otherwise, those who selected the first or second options were defined as “current Mormons” in our survey and those who selected the third option as “former Mormons.”

The data collection ended with a final distribution of: 943 native Mormon (55.6% of total), 213 convert Mormons (12.6 percent of total), and 540 former Mormons (31.8 percent of total) with fully completed surveys.

Other Survey Parameters

Age: A primary objective of the NMS is to compare religious behaviors and attitudes across generational lines, so we established age quotas to ensure that we would obtain a representative sample of each generation given that internet surveys tend to bias toward younger respondents. Our quotas were based on the age distributions as given in the 2007 Pew Religious Landscape Study, so that the NMS would mirror the wider LDS population as much as possible. (The 2014 Pew Religious Landscape Study would have been more ideal, but at the time of the survey certain essential portions had not yet been released for public analysis.) At the end of the data collection, the NMS sample had finished at:

• 42 percent Millennials (born 1980 to 1998)
• 30 percent GenXers (born 1965 to 1979)
• 28 percent for the Baby Boomers (born 1945 to 1964) and Silent Generation (born 1928 to 1944) combined together.

We use these three principal age cohorts throughout the book. As we will see below, because the NMS slightly oversampled younger respondents, we corrected for that with a post-survey weighting procedure.

Gender: We originally did not include any quotas by way of gender on the assumption that men and women would respond to the survey roughly in proportion to their distribution in the wider population of interest. It became apparent as data commencement began, however, that more women were responding to the survey than men. Despite implementing a more stringent quota, the survey ended with a final sample of current/former Mormons that was 36.4 percent male, 63.3 percent female, and 0.4 percent “other.” This necessitated a weighting of the sample, as described below, to achieve representativeness for gender so that men’s opinions would be weighted according to men’s actual percentage of the LDS population, which is slightly less than half.

Geography: We requested that 30 percent of the sample of current and former Mormons report currently living in Utah and 70 percent outside of Utah, again based on the geographical distribution revealed in the 2007 Pew Religious Landscape Study. We wanted to ensure to the extent possible that our results did not describe Utah Mormons only. In all, 27 percent of our final sample of current/former Mormons reported currently living in Utah, 15 percent in the Mountain states, 12 percent in the Midwest, 6 percent in the Northeast, 18 percent in the Pacific states, 16 percent in the South Atlantic, and 6 percent in the South Central states, still very close to our goal distribution based on Pew data.

Assessing the Survey’s Representativeness

After data collection was complete for all 1,696 respondents, we compared our survey results with those of other reputable and nationally representative public opinions surveys based on random-digit dialing methods to assess the degree of representativeness that we achieved with the NMS. Specifically, we compared our results with those of the 2014 Pew Religious Landscape Study (which had been released just after the NMS’s data collection process) on several demographic and socioeconomic indicators such as gender, income, education, age, and race/ethnicity.

Table 1. Demographic variables of the NMS before weighting

	CURRENT MORMONS – 2014 RLS (WEIGHTED) MoE ±4%	CURRENT MORMONS – 2016 NMS (UNWEIGHTED) MoE ±3%	DIFFERE NCE		FORMER MORMONS – 2014 RLS (WEIGHTED) MoE ±7%	FORMER MORMONS – 2016 NMS (UNWEIGHTED) MoE ±4%	DIFFERENCE
Male	46%	39%	-7%		55%	32%	-23%
Female	54%	61%	7%		45%	67%	22%
Less than college degree	67%	55%	-12%		79%	64%	-15%
College	21%	31%	10%		14%	25%	11%
Postgraduate	12%	14%	2%		8%	11%	3%
Income < $50K	47%	47%	0%		51%	54%	3%
Income $50K-$100K	33%	37%	4%		35%	32%	-3%
Income > $100K	20%	45%	-3%		14%	14%	0%
Millennial	31%	44%	13%		40%	38%	-2%
GenXer	31%	27%	-4%		32%	37%	5%
Boomer/Silent	38%	29%	-9%		28%	25%	-3%
White	85%	86%	1%		81%	84%	3%
Black	1%	4%	3%		3%	5%	2%
Hispanic	8%	5%	-3%		5%	6%	1%
Other race/ethnicity	6%	5%	-1%		12%	6%	-6%
Utah	28%	29%	1%		21%	24%	3%
Non-Utah	72%	71%	-1%		79%	76%	-3%

Note: Categories may not sum to 100 percent due to rounding. The 2014 Pew Religious Landscape Survey samples of comparison include 664 self-identified Mormons and 215 self-identified former Mormons.

Table 1 shows us that for the current Mormon sample, our internet-based NMS achieved a level of demographic and socioeconomic representativeness very close to that of the random-digit dialing telephone-based RLS survey with a few moderate exceptions. Specifically, the NMS oversampled women compared to men by about 7 percent. It undersampled those with a high school education by about 10 percent and oversampled those with a college degree by the same amount. Millennials were oversampled compared to Baby Boomers and Silent Generation members. In terms of income, race/ethnicity, and geographical residence, however, the two surveys were virtually identical and certainly within the margin of sampling error.

Table 1 also shows us that for our former Mormon sample, the NMS survey did an even better job of approximating demographic and socioeconomic distributions in the wider population (as indicated by the 2014 RLS survey). The only two categories where the NMS survey differed appreciably was that it substantially oversampled former Mormon women compared to men and those with a college degree compared to those with a high school education.

It is important to note that these sampling differences are extremely common in public opinion survey research. When this happens, researchers can create “post-stratification sample weights” which minimize potential biases in the survey results due to disproportionate sampling of one group over another. In other words, we can statistically correct for these sample biases to a large extent by artificially inflating the weight of the responses from groups that were undersampled in the survey while artificially contracting the weight of the responses from the groups that were oversampled, in direct proportion to the degree to which they were over- or undersampled in the survey. Assuming that there are correlations with the particular survey question and a demographic or socioeconomic factor like age, education, gender, and so forth, this procedure increases our confidence that our survey findings are representative of the wider population of interest.

For the purposes of this research project, we created two separate sample weights, one for current Mormons and one for former Mormons, to correct for these survey biases. The sample weight for current Mormons accounts for gender, education, and age while the sample weight for former Mormons accounts for gender and education. We used the weighted 2014 Pew Religious Landscape Study as our basis of comparison when constructing our sample weights. Unless otherwise noted, these sample weights were applied for all analyses reported.

Table 2. Demographic variables of the NMS after weighting

	CURRENT MORMONS – 2014 RLS (WEIGHTED) MoE ±4%	CURRENT MORMONS – 2016 NMS (WEIGHTED) MoE ±3%	DIFFERE NCE		FORMER MORMONS – 2014 RLS (WEIGHTED) MoE ±7%	FORMER MORMONS – 2016 NMS (WEIGHTED) MoE ±4%	DIFFERENCE
Male	46%	47%	1%		55%	54%	-1%
Female	54%	53%	-1%		45%	44%	-1%
Less than college degree	67%	65%	-2%		79%	79%	0%
College	21%	22%	1%		14%	13%	-1%
Postgraduate	12%	12%	0%		8%	8%	0%
Income < $50K	47%	48%	1%		51%	55%	4%
Income $50K-$100K	33%	36%	3%		35%	31%	-4%
Income > $100K	20%	16%	-4%		14%	14%	0%
Millennial	31%	35%	4%		40%	33%	-7%
GenXer	31%	35%	4%		32%	37%	5%
Boomer/Silent	38%	30%	-8%		28%	30%	2%
White	85%	87%	2%		81%	83%	2%
Black	1%	4%	3%		3%	6%	3%
Hispanic	8%	4%	-4%		5%	5%	0%
Other race/ethnicity	6%	4%	-2%		12%	6%	-6%
Utah	28%	28%	0%		21%	22%	1%
Non-Utah	72%	72%	0%		79%	76%	-3%

With these data weights applied, our survey results become even more representative of both current and former Mormons on almost every demographic category, as shown in Table 2. They are often nearly identical to those of the 2014 Pew Religious Landscape Study (or well within the margin of sampling error). We note as well that the majority of our weighted survey results for key religious and political attitudes/behaviors also approximate those found for current and former Mormons in the 2014 Pew Religious Landscape Study. We thus argue that the results we report in this book are representative of the wider Mormon and former Mormon populations in the United States within the standard margins of error (±3 percent and ±4 percent, respectively) for public opinion survey research.

Survey Question Wording

In designing a survey with over 130 questions (and more than 600 separate variables), we consulted a variety of sources and questionnaires from other religious and social surveys conducted by reputable research firms, including those who have specifically surveyed Mormons. In particular, many survey questions came from standard batteries of survey questions on religion and social behavior used by the Pew Religious Landscape Study, the General Social Survey, the American National Elections Survey, the 2012 Pew Mormons in America Survey, the Pew Political Typology surveys, the Peculiar People Survey, the 2011 Understanding Mormon Disbelief Survey, and the 2015 Mormon Gender Issues Survey. Many other questions we wrote ourselves according to the goals of our research project, striving in each case to follow best practice public opinion survey question design. After the original survey was drafted, we sought input from other scholars and experts of religion, public opinion, and Mormonism. Many people graciously contributed constructive feedback that further strengthened the survey. These include social scientists David Campbell, Ryan Cragun, Armand Mauss, Gary Shepherd, Gordon Shepherd and Rick Phillips, as well as other individuals who are listed in the acknowledgments. Angela Black, Matthew Bowman, Joanna Brooks, Tona Hangen, Sharon Harris, Tania and John Lyon, Allison Pond, Bob Rees, Travis Stratford, Toria Trendler, Bill and David Turnbull, and Sara Vraneš also offered careful and conscientious feedback as we improved the questions through eighteen different survey drafts before fielding the survey. Throughout the book the survey question wordings are provided in the text and are footnoted if otherwise. The entire final survey is also available here on this website as a downloadable PDF.