4.2 Estimation of proportions
Proportions express the relative weight of specific groups within the population. For example, knowing the percentage of households below the poverty line is essential for assessing inequalities and characterizing well-being conditions. To obtain this type of indicator, the weighted mean of the dichotomous variable is calculated, ensuring that the estimate adequately represents the population distribution. According to Heeringa et al. (2017), when the original categories are converted into indicator variables, the estimated proportion is calculated as:
\[ \hat{p}_d = \frac{\hat{N}_d}{\hat{N}} = \frac{\displaystyle\sum_{h} \sum_{i} \sum_{k} w_{hik} \ I(y_{hik}=d)} {\displaystyle\sum_{h} \sum_{i} \sum_{k} w_{hik}} \]
Although this estimator is conceptually simple, it is a nonlinear function of population totals. For this reason, deriving its statistical properties, particularly its variance and standard error, requires approximation methods such as Taylor linearization. In this context, the function \(z_{hik}=I(y_{hik}=d)-\hat{p}_d\) is used. In practice, statistical software implements these procedures automatically and produces proportion estimates together with their corresponding standard errors and confidence intervals.
When proportions take values very close to 0 or 1, the precision of the estimates may be affected, so in many cases larger sample sizes are needed to obtain stable and reliable results. This situation is especially relevant in the analysis of rare or highly concentrated phenomena, where small variations in the sample can produce important changes in the estimates.
In addition, confidence intervals constructed using normal approximations may have lower limits below 0 or upper limits above 1, making them difficult to interpret because they are proportions. To avoid this problem, a widely used alternative is to apply the logit transformation before constructing the confidence intervals. This procedure ensures that, after returning to the original scale, the limits obtained remain within the valid range for proportions.
\[ CI(\hat{p}_d; 1-\alpha) = \frac{exp \left[ln\left(\frac{\hat{p}_d}{1-\hat{p}_d}\right) \pm \frac{\widehat{me}}{\hat{p}_d(1-\hat{p}_d)}\right]}{1 + exp \left[ln\left(\frac{\hat{p}_d}{1-\hat{p}_d}\right) \pm \frac{\widehat{me}}{\hat{p}_d(1-\hat{p}_d)}\right]} \]
Where \(\widehat{me}\) corresponds to the estimated margin of error and is obtained as the product of the critical value from Student’s \(t\) distribution and the estimated standard error of the proportion \(\hat{p}_d\). The term \(t_{1-\alpha/2, df}\) represents the percentile associated with the selected confidence level, considering \(df\) degrees of freedom. In the context of complex surveys, these degrees of freedom are usually defined as the difference between the number of primary sampling units (PSUs) and the number of strata present in the subgroup or analysis domain.
Unlike other statistical software that presents results as percentages, R reports proportions on the [0,1] scale. The functions in these packages make it possible to calculate the estimates, their standard errors, and confidence intervals directly, facilitating the analysis of categorical variables in household surveys. The following example illustrates how to obtain the proportion of people by zone using the sampling design defined previously, presented in Table 4.5:
survey_design %>%
group_by(Zone) %>%
summarise(proportion = survey_mean(
vartype = c("se", "ci"),
proportion = TRUE
))| Zone | proportion | proportion_se | proportion_low | proportion_upp |
|---|---|---|---|---|
| Rural | 0.48 | 0.014 | 0.452 | 0.508 |
| Urban | 0.52 | 0.014 | 0.492 | 0.548 |
In this example, the survey_mean() function is used to calculate the estimate of population proportions and, through the proportion = TRUE argument, it specifies that the analysis focuses on estimating the relative distribution of the population across the different categories of the variable analyzed. The results indicate that approximately 47.9% of people live in the rural zone, with a 95% confidence interval between 45.2% and 50.7%, while 52.0% corresponds to the population living in the urban zone, with a confidence interval between 49.2% and 54.7%.
The survey library also provides the survey_prop() function, designed specifically for estimating proportions in complex surveys. This function produces results equivalent to those obtained with survey_mean(), while also offering a more direct and intuitive syntax when the objective of the analysis is exclusively to calculate proportions. The results obtained with this alternative are presented in Table 4.6.
These estimates show that, in the urban zone, 53.6% of the population is female, with a 95% confidence interval between 51.0% and 56.2%, while 46.4% is male, with a confidence interval between 43.7% and 48.9%. These results make it possible to examine not only the percentage distribution of the population by sex, but also the level of precision associated with the estimates.
| Zone | proportion | proportion_se | proportion_low | proportion_upp |
|---|---|---|---|---|
| Rural | 0.48 | 0.014 | 0.452 | 0.508 |
| Urban | 0.52 | 0.014 | 0.492 | 0.548 |
If the focus is now on estimating proportions for specific subpopulations, for example, the proportion of men and women living in the urban zone, the analysis must be restricted to that study domain and the corresponding estimates then calculated for each sex category. The corresponding computational code is presented in Table 4.7.
| Sex | proportion | proportion_se | proportion_low | proportion_upp |
|---|---|---|---|---|
| Female | 0.537 | 0.013 | 0.511 | 0.563 |
| Male | 0.463 | 0.013 | 0.437 | 0.489 |
Similarly, the same exercise can be carried out for the population living in the rural zone by estimating the proportion of men and women. This type of disaggregation makes it possible to compare the population composition by sex between urban and rural areas, providing a more detailed view of the demographic characteristics of the surveyed population. The corresponding results are presented in Table 4.8.
rural_subset %>%
group_by(Sex) %>%
summarise(
n = unweighted(n()),
proportion = survey_prop(vartype = c("se", "ci"))
)| Sex | n | proportion | proportion_se | proportion_low | proportion_upp |
|---|---|---|---|---|---|
| Female | 679 | 0.516 | 0.008 | 0.500 | 0.533 |
| Male | 618 | 0.484 | 0.008 | 0.467 | 0.500 |
Now, if the analysis is restricted only to the male population included in the database and the interest is to estimate the percentage distribution of men by zone of residence, it is possible to calculate the proportion of men living in urban and rural areas using the sampling design defined previously. The corresponding computational code is presented in Table 4.9.
| Zone | proportion | proportion_se | proportion_low | proportion_upp |
|---|---|---|---|---|
| Rural | 0.491 | 0.018 | 0.455 | 0.526 |
| Urban | 0.509 | 0.018 | 0.474 | 0.545 |
If results with several levels of disaggregation are to be estimated, as in previous chapters, the use of the group_by() function makes it possible to generate different aggregation levels by combining two or more categorical variables. For example, it is possible to estimate the proportion of men by zone of residence and poverty status simultaneously. This type of analysis makes it possible to characterize the socioeconomic conditions of specific subpopulations in greater detail. The corresponding procedure is presented in Table 4.10.
male_subset %>%
group_by(Zone, Poverty) %>%
summarise(proportion = survey_prop(vartype = c("se", "ci")))| Zone | Poverty | proportion | proportion_se | proportion_low | proportion_upp |
|---|---|---|---|---|---|
| Rural | NotPoor | 0.549 | 0.063 | 0.424 | 0.668 |
| Rural | Extreme | 0.198 | 0.067 | 0.096 | 0.364 |
| Rural | Relative | 0.254 | 0.037 | 0.187 | 0.334 |
| Urban | NotPoor | 0.660 | 0.037 | 0.584 | 0.728 |
| Urban | Extreme | 0.113 | 0.025 | 0.073 | 0.171 |
| Urban | Relative | 0.227 | 0.026 | 0.180 | 0.283 |
Based on the results obtained, 19.7% of men in the rural zone are living in extreme poverty, while in the urban zone this proportion is 11.3%. Although the point estimates suggest a higher incidence of extreme poverty in rural areas, comparison of the confidence intervals indicates that there is not enough evidence to conclude that the differences are statistically significant at the confidence level considered, because the two intervals overlap.
It is also possible to categorize quantitative variables to facilitate their joint analysis with other categorical variables. For example, age can be grouped into specific age ranges and then crossed with variables related to employment status. This type of procedure is useful for identifying differences in labor participation across age groups.
Below, the age variable is classified into two categories for the female population: women between 18 and 35 years of age, and women belonging to other age ranges. These categories are then analyzed together with the employability variable, making it possible to estimate the distribution of labor status by age group. The corresponding results are presented in Table 4.11.
female_subset %>%
filter(!is.na(Employment)) %>%
mutate(age_range = case_when(
Age >= 18 & Age <= 35 ~ "18 - 35",
TRUE ~ "Other"
)) %>%
group_by(age_range, Employment) %>%
summarise(proportion = survey_prop(vartype = c("se", "ci")))| age_range | Employment | proportion | proportion_se | proportion_low | proportion_upp |
|---|---|---|---|---|---|
| 18 - 35 | Unemployed | 0.029 | 0.009 | 0.015 | 0.054 |
| 18 - 35 | Inactive | 0.517 | 0.038 | 0.442 | 0.591 |
| 18 - 35 | Employed | 0.455 | 0.036 | 0.385 | 0.526 |
| Other | Unemployed | 0.016 | 0.007 | 0.007 | 0.036 |
| Other | Inactive | 0.571 | 0.030 | 0.511 | 0.629 |
| Other | Employed | 0.413 | 0.029 | 0.356 | 0.472 |