Moving towards a single-frame cell phone design in random digit dialing surveys: considerations from a French general population health survey | BMC Medical Research Methodology

The French Health Barometer is a repeated health telephone survey that has been conducted in the general French population since 1992 by Santé publique France, the national public health agency, allowing trends in health risk behaviors to be measured. To meet the challenge of adequately covering the population, the sampling design of the Health Barometer survey has regularly evolved. From 1992 to 2005, it used a SF directory-based survey of landline phone numbers. From 2005 to 2014, the survey used a screening approach with a disjoint DF sample of landline telephones and cell phone-only owners. Since 2014, an overlapping DF design of landline and cell phone numbers has been implemented using RDD [26]. Considering the recent evolutions in telephone usage, we question whether it is time to stop contacting people on landlines. It is therefore necessary to evaluate the impact of such a change on our estimates.

  1. 1.

    Health Barometer data and design weight

The 2017 French Health Barometer survey used the RDD method, and included a respondent sample of 25,319 people aged 18 to 75 years, living in France, non-institutionalized, and speaking French. Interviews were conducted using computer-assisted telephone interviewing. Participation was anonymous and voluntary. In accordance with the guidelines of the French Data Protection Authority (Commission Nationale de l’Informatique et des Libertés, CNIL), all subject included in this study gave their informed verbal consent to participate before the telephone interview.

The Health Barometer survey 2017 used an overlapping DF design of landline and cell phone numbers. For cell phones, the selected interviewee was the person who picked up the phone. For landline phones, the Kish selection method was used: it consists in listing all household members and then drawing randomly the person to interview among the eligible persons, with the same probability for each eligible person [31]. Thus, the sampling design was one-stage for cell interviews and two-stage for landline interviews. Hereafter, the sampling frame is the list of generated telephone numbers.

Design weights, reflecting the individual selection probability, were calculated for the DF sample (df) using information about the number of phone numbers generated, the number of phone numbers owned by the respondent (reported in the questionnaire), and the number of eligible persons in the household for landlines. Design weights are the inverse of the individual selection probability \({\pi}_i^{df}\), defined as:

$${\pi}_i^{df}=\frac{n_{LL}}{N_{LL}}\ast \frac{t_{LL}^i}{e_{LL}^i}+\frac{n_C}{N_C}\ast \frac{t_C^i}{e_C^i}$$

where i denotes an eligible person; NLL and NC denote, respectively, the number of landline and cell phone numbers in the sampling frame; nLL and nC denote, respectively, the number of landline and cell phone numbers in the sample; \({t}_{LL}^i\) and \({t}_C^i\) denote, respectively, the number of landline and cell phone numbers leading to contact with person i; \({e}_{LL}^i\) denotes the number of eligible persons per household for the landline number i, and \({e}_C^i\) denotes the number of eligible individuals who use the cell number \(i\ \left({e}_C^i=1\right)\).

  1. 2.

    Counterfactual cell-phone only design weight

For the purpose of this study, we created an “as-if” SF cell phone sample (sf) by selecting cell phone respondents of our survey, i.e. by excluding the landline phone respondents from the DF sample. These weights, computed for the cell phone respondents only, are the inverse of \({\pi}_i^{sf}\), with:

$${\pi}_i^{sf}=\frac{n_C}{N_C}\ast \frac{t_C^i}{e_C^i}$$

  1. 3.

    Calibrated weights

The design weights (DF and SF) were then calibrated to adjust to the French population structure as reported by the Labor Force Survey (conducted by the French National Institute for Statistics and Economic Studies, INSEE) using the raking ratio [32]. The calibration covariates were gender by age, education level, size of household, urbanization, and region of residence. Note that these variables include age and education that are prominent determinants of health behaviors [33] (see Table 1 for details).

  1. 4.

    Impact of a SF cell phone design

Table 1 Distribution of calibration covariates in the reference population and standardized distances

To evaluate the impact of moving to a SF cell phone design in terms of effectiveness and related costs, we first compared the performances of landline and cell phone interviews, using productivity criterions (interview duration, call attempts and generated phone numbers required to obtain an interview) and response rates. The original disposition codes used to calculate the response rates were mapped to the specified codes and formula #3 of the American Association for Public Opinion Research (AAPOR) (The American Association for Public Opinion Research, 2016, Survey Outcome Rate Calculator 4.0.)

Second, we studied the representativeness of the current dual-frame design (DF) and the counterfactual single-frame (SF) cell phone design, by using the corresponding design weights defined above. For this purpose, we studied first the balance of the effective DF and of the counterfactual SF samples to the population reference margins for each of the calibration covariates. The reference margins of the population for calibration covariates were taken from the 2016 Labor Force Survey conducted by INSEE, restricted to the same age-range (18–75 years old). The representativeness of each sample (DF and SF) was then also assessed for several external covariates using the corresponding calibrated weights. These external covariates were socio-demographic characteristics (occupational status and socio-professional group), and a health-related outcome (visits to a family physician in the preceding year), for which a gold-standard was available. These gold standards were taken from the 18–75 years old population in the 2016 Labor Force Survey for employment status and socio-professional group, and from National Health Insurance data for visits to a family physician in the preceding year. To compare samples in a univariate approach, we used standardized distances d [34]. For each variable category, a dcategory was calculated as in formula (1) with pA the prevalence in the reference population; pB the weighted prevalence estimated in the sample; qA = 1 − pA; qB = 1 − pB. The information was then summed up at the variable level using a mean of absolute values of d on all m categories. Finally, at the sample level, we summed up the distances with a mean of all Dvariable. Following Austin’s recommendations [34, 35], we considered a value below 10% as reflecting an acceptable balance.

$${d}_{category}=100\times \left({p}_B-{p}_A\right) \left/ \sqrt{\left({p}_A{q}_A+{p}_B{q}_B\right)\left/ 2\right.}\right.$$


To provide a multivariate insight into the global distribution of a sample, we used R-indicators [36], which were based on a response propensity model. R-indicators estimate the variation of the predicted probability of belonging to the studied sample instead of the benchmark sample, conditionally to the variables included in the model. R varies between 0 and 1; the higher the value, the better the balance conditionally to all variables. We computed a model without interactions. The 2017 annual census, which collected data on 6,162,026 individuals aged 18–75 years living in non-institutionalized households, was used as the reference population to compute R-indicators.

In a third step, we analyzed the impact of switching to a SF cell phone design for seven health indicators: self-reported health status as “poor,” chronic diseases, limitations in daily activities, obesity (body mass index ≥30), physical inactivity (physical activity of at least 30 min less than once a month), daily cigarette smoking, and lifetime suicidal attempts. The first three indicators compose the Minimum European Health Module (MEHM), which is a standardized instrument used to monitor the different dimensions of health at national and international levels [37]. For the mental health dimension, we chose lifetime suicidal attempts as France has one of the highest suicide rates in Europe [38]. Finally, tobacco smoking and obesity were studied as important risk factors for a number of non-communicable diseases, including cancer, diabetes and cardiovascular disease [39, 40]. These risk factors were described in previous studies to be inequitably distributed across the different socio-economic strata, that is obesity and tobacco smoking are concentrated in the least affluent and the least educated parts of the population, contributing to increase the social inequalities in health [41,42,43].

We provided prevalence estimates calculated for the combined DF sample and the counterfactual SF sample using the corresponding calibrated weights. The health indicators were estimated for the whole population, and also for two age subgroups (18–30 year-olds and 60–75 year-olds) being respectively the most and the least equipped with cell-phones. To quantify the possible impact on the prevalence estimates, we used the relative difference between the combined DF sample \(({\overline{y}}_{df})\) and the SF sample \(({\overline{y}}_{sf})\), defined as:

$$Relative\ Difference=\frac{{\overline{y}}_{sf}-{\overline{y}}_{df}}{{\overline{y}}_{df}}$$

The sampling design impact was evaluated using the deft indicator, defined as the square root of the design effect for each health indicator:

$$deft=\sqrt{\frac{\mathit{\operatorname{var}}\left(\overline{y}\right)}{s^2\left/ n\right.}}$$

where \(\mathit{\operatorname{var}}\left(\overline{y}\right)\) is the variance observed for y on the sample, s2 is the variance estimated under a simple random sampling, and n is the sample size [44]. s2 equals p(1-p) when y is a binary variable of proportion p. Deft shows how much the survey design impacts the sample standard error (SE), and consequently, the confidence intervals.

Calibrated weights were also compared between the combined DF sample and the SF sample using the max/min ratio and the coefficient of variation, calculated as the ratio of the standard error over the mean.

Analyses were conducted using SAS 9.4 and Stata 14.2. Syntax used in the data analysis are provided in Supplementary materials. The data are available on request from the corresponding authors.

Source link

Leave a Comment