Use of Text Messaging to Increase Response Rates

Piper DuBray

ICF International 126 College St. Burlington, VT 05401

Proceedings of the 2013 Federal Committee on Statistical Methodology (FCSM) Research Conference

Introduction

Survey response rates have greatly declined in the past decade, causing researchers to seek new ways to increase

participation. The Connecticut Department of Health (CT DPH) and ICF International conducted two pilot studies in

2012 using text messages to 1) Increase response rates to the Behavioral Risk Factor Surveillance System (BRFSS)

Cell phone Survey, and 2) Increase participation in the BRFSS Non-Response Web Follow-up Survey.

The Centers for Disease Control and Prevention (CDC) established the state-level BRFSS in 1984. By 1994, all

states and US territories are participating. In order to increase response to the landline survey, many states have

adopted the use of advance letters to notify landline respondents whose phone numbers have been matched to an

address of the upcoming BRFSS call. Cell Phone numbers are unlisted and therefore cannot be matched to

addresses. As a result, cell phone respondents do not receive advance letters, nor do they receive the mailed

invitation to participate in the BRFSS Web Survey.

Research Questions

The purpose of our research was to explore the effects of sending text messages to potential cell telephone and web

survey participants. We wanted to determine whether sending a text message to cell phone sample in advance of

upcoming calls increases response to the cell phone survey; also, whether sending a text message to cell phone non-

responders inviting them to complete the Web Survey increases web survey participation. Finally, we also examined

whether the respondents who participate after receiving a text message any different from other respondents.

Methods – Using Text Messages as Advance Notification to Cell Survey

For our first experiment, we looked at sending text messages as advance notification for a cellular telephone

interview. In this experiment, we divided cell phone sample from October 2012-March 2013 into 3 groups. Group 1

was sent a text notification that included an offer of a ten dollar incentive, provided in cash by mail or an immediate

Amazon.com gift code after completing the telephone interview. Group 2 was sent a text notification without the

offer of an incentive. Group 3 was the control group and did not receive any text notification.

The text messages were sent shortly after sample was received from CDC each month, prior to any attempts being

made by the CATI center. Approximately 24 hours after the text was sent, ICF reviewed responses and removed any

upset respondents from the phone study as a refusal.

The CATI program was modified to add questions to the end of the interview about the text message and to collect

information for incentives. Questions added to CATI survey for Groups 1 and 2:

1. Prior to our call today, we sent a text message to potential cell phone survey participants. Did you receive a text

message about this interview?

1 Yes

2 No

7 Don’t Know

9 Refused

If Group = 1 (incentive) and Q1 = 1 (recall text), Q2 was asked.

Q2. In appreciation for the time you have spent answering our questions, we would like to provide you with $10 in

compensation. Would you like to give us your address so that we can mail you the payment, or would you rather I

give you a $10 Amazon.com gift code now?

01 Agree to give address information

02 Amazon.com gift code

99 Refused $10

Results - Using Text Messages as Advance Notification to Cell Survey

In this first experiment, a total of 35,460 text message notifications were sent. Of the messages sent, 73 respondents

(0.2%) replied to the message to opt out of the survey.

Table 1: Sample Size

shows the sample size for each pilot

group.

Pilot Group Sample

1. Advance text with Incentive 11,492

2. Advance text without Incentive 11,666

3. No advance text 12,302

Table 1: Sample Size

In response to the question in the CATI survey, the method of incentive preferred was the gift code. The results are

shown in

Table 2: Preferred Incentive Mode

Incentive Mode Percent

Preferred Cash Incentive 44%

Preferred Gift Code 53%

Refused Incentive 3%

Table 2: Preferred Incentive Mode

The portion of the sample that was sent a text notification including an offer of an incentive, or Pilot Group 1, had

the highest response and cooperation rates; however none of the differences were significant. Table 3: Response

Rates shows the BRFSS Cooperation rate, Refusal Rate, and CASRO Response Rate for each pilot group.

Pilot Group Cooperation Rate

Refusal Rate

CASRO Rate

1. Advance text with

Incentive

63.0% 9.1% 23.4%

2. Advance text without

Incentive

61.8% 8.6% 23.0%

3. No advance text 59.4% 10.4% 22.5%

Table 3: Response Rates

Next, we compared demographic data between respondents in each pilot group. Income stood out as a significant

(chi-sq=7.16, p=.0075)

association with the text including incentive group (Group 1). Sending the notification with

the incentive provided an increased response in the highest income category, while the lower income categories

were more likely to have a higher response from the no-text group (Group 3). In the highest income category, 9%

more respondents were from the group that was sent the notification including the incentive, than group 3, the no

text group. In the income $35,000 < $50,000 category, 8% more respondents were from the no text group (Group 3)

than the text with incentive group (group 1).In the income $25,000 < $35,000 category, 6% more respondents were

from the no text group (Group 3) than from the text with incentive group (Group 1). These results are shown in

Figure 1.

< $10k < $15k < $20k < $25k < $35k < $50k < $75k $75k or

Percent Respondents in Pilot Group

Income Category

Text

With

Incentive

Text

Without

Incentive

No Text

Figure 1 Income Category by Pilot Group

During the BRFSS CATI interview, we asked respondents if they recalled receiving the text message. Of the two

pilot groups that were sent text notifications, the recall of the text message was about equal. Overall, a third of the

respondents recalled receiving the text message, however these differences were not significant, (P=.7377). It is still

unclear from these results if those not recalling the notification simply do not remember it, or if they did not receive

the message.

Text Group

Recall Getting

the Text

Message

Did Not Recall

Getting Text

Message

DK if Received

Text Message

Refused

Advance text with

Incentive

35% 55% 9% 1%

Advance text

without Incentive

31% 58% 10% 1%

Total 33% 56% 10% 1%

Table 4: Text Recall

Methods – Using Text Messages as an Invitation to Web Survey

In our second experiment, we reviewed the use of text messages as invitations to participate in an online survey. In

2012, CDC conducted a Web Pilot for the BRFSS Survey, in which landline survey non-responders received a letter

inviting them to complete the Web survey. During August and September of 2012, ICF conducted a pilot in which

we sent a text message to non-responders of the CT BRFSS Cell Phone Survey, inviting them to complete the Web-

based BRFSS survey. All text message invitations also included an option to opt out of future messages.

Approximately one week after the first text message was sent, a reminder text was sent, excluding any respondents

who opted out.

The text message invitations were divided into two pilot groups. Group 1 was sent a text invitation that offered a $10

Amazon.com gift code for completing the survey.

Text 1 (160 characters): Participate in the CT Dept. of Public Health & CDC Study, receive $10 gift code

https://www.healthdepartmentsurvey.org/ Login ID CT8xxxxx Reply STOP to opt out.

Text 2 (132 characters): As a thank you, we will send you a text containing a $10 Amazon.com gift code

approximately 4-8 weeks after you complete the survey.

Group 2 received the same text invitation, however with no incentive offer.

No Incentive Group (136 characters): Participate in the CT Dept. of Public Health & CDC Study

https://www.healthdepartmentsurvey.org/ Login ID CT7xxxxx Reply STOP to opt out

One week after the first invitation was sent, the same text was sent as a reminder to all who had not opted out.

Incentive delivery text: Thank you for participating in the CT Dept. of Health and CDC study. Your $10

Amazon.com gift code is: xxx-xxxxxx-xxxx.

Results - Using Text Messages as an Invitation to Web Survey

During the two month experiment, a total of 4,847 text invitations were sent to BRFSS cell phone non-responders.

Nine percent, or 425 respondents, replied “Stop” to opt out of the web invitation. Only 36 respondents participated

in the web survey after being sent the text invitation.

Table 5

shows the number of invitations sent in each group and

the percent of those which completed the web survey.

Text Group

Text Invitations Sent

Web Surveys Completed

Percent Completed

Group 1: With Incentive 2,424 26 1.1%

Group 2: Without Incentive 2,423 10 0.4%

Total 4,847 36 0.7%

Table 5: Web Survey Participation after Text Invitation.

The text message invitations produced a much lower response than the mail invitations that are sent to the landline

BRFSS non- responders. The invitations that were mailed to the landline sample yielded 263 completed web

surveys, a completion rate of 8.2%.

Mail Invitations Sent

Web Surveys Completed

Percent Completed

3,208 263 8.2%

Table 6: Web survey participation after mailed invitation.

Demographic data was then compared between web respondents that were recruited from the landline non-

responders by mail, and those recruited from the cell phone non-responders via text messages. There were two

demographic areas, marital status and age, that stood out as having significant associations between the mail and text

recruitment modes. More non-married completes, 31%, came from the group recruited by text with incentive, then

either the text without incentive or mailing groups. However, only the mail vs. both text recruitment groups

combined had a significant difference (P=0.0091). The two incentive groups within the text pilot group did not show

significance (P=0.1121) however this is likely due to having such a small sample size in the text group.

The group recruited from the cell phone non-responders via text-message had 38% more 18-34 year olds as

compared to the group recruited by mail. (P<0.001)

Figure 2

shows the % of respondents in the web survey that

were married and the % that were age 18-34, by mode of recruitment.

Figure 2: Web Survey Demographics by Land and Cell Recruitment Mode

67%

44%

47%

10%

20%

30%

40%

50%

60%

70%

80%

Married Age 18-34

Land

Cell

Our last comparison looked at the demographic characteristics of respondents from the cell phone sample, between

those who completed the cell telephone survey, and those who completed the survey via web. Education was the

only category showing a significant difference between modes of completion. Within the cell sample respondents

who completed the survey on the web, 85% had some college or higher level education; only 66% of cell sample

respondents that completed on the phone had some college or higher level education.

Results – Costs of Using Text Messaging.

Sending text message advance notifications to cell phone respondents increased the cell phone cost per complete by

3.8%. This considers the time and effort of sending the text messages and effort used to remove the opt-out records

from the cell phone survey sample.

Sending text message invitations to the web survey to cell phone non-responders increases the cost for web survey

preparation by 27.7% over the cost of sending a mailed invitation to landline sample. The cost considers a 2-month

fielding for both the mail and text web follow up; the mailing consisting of an initial letter followed by a reminder

postcard, batched and sent every two weeks; the text invite consisting of an initial text followed by a reminder

text, batched and sent every two weeks.

Conclusions - Advance Text Message and Cell Survey Response

We saw some increases in response, but none were statistically significant. Text messages with and without

incentives are an option for improving cell response, but alone may not provide a significant increase. It may be

worthwhile to consider text message notification and incentives, combined with other efforts to increase cell

response. There were some between group differences on income. Texts with and without incentives may be

considered if specific income groups are being targeted.

Conclusions – Text Invitations to Web Survey

A small number of completes were obtained through the text invitation, resulting in a total increase in Web

participation of 13.7%. Using text message invitations means participation from people who otherwise would not

have been reached. The additional cost required to send the text messages is considerable compared to the low

number of completes that were obtained.

There was a between group difference on marital status. Invitations by text

message with incentives may be used to increase response from unmarried respondents. Also, recruiting Web

respondents via cell phone produced a higher number of respondents with at least some college education.

Limitations

These experiments with using text messages taught us that some carriers can block our outgoing number without us

knowing. Therefore, we can only assume the text messages that we are sending out are being delivered by each

carrier. We attempted to measure this by using a question of recall on those who did respond to the survey, which

showed a 33% recollection rate of the text message. This could mean that 2/3 of our outgoing texts were not

delivered.

In the second experiment that used text messages as an invitation to the Web survey, the survey itself was not

created to be viewed on smartphones. We may have lost some participation from respondents that could not view

the survey directly on their phone.

Sending the advance text messages to cell phones had an effect on the CATI center schedule, as we had to delay the

start of calling monthly sample while we sent text messages and gave 24 hours to remove any who replied “stop” to

opt out of the survey.

Piloting additional studies with more data collection time points and the possibility of a larger overall incentive

would be helpful in determining how best to implement this type of data collection effort and retain respondents

over time.