3.2 Hypothesis testing
In the analysis of data from household surveys, it is not enough to study each variable in isolation, such as estimating the average income of men and women in a country. It is also essential to compare groups and evaluate whether the differences observed between them reflect real inequalities in the population or if they could be explained simply by sampling error. For example, a frequently asked question is whether there are statistically significant differences in median income between male-headed households and female-headed households. This type of analysis makes it possible to identify social and economic gaps, in addition to generating useful evidence for the design and evaluation of public policies.
To answer these types of questions, hypothesis tests are used, that is, statistical procedures that allow statements about population parameters to be contrasted using the information observed in the sample. In the context of household surveys, these tests must adequately incorporate the characteristics of the sampling design in order to guarantee valid inferences and reliable results.
Hypothesis testing is a fundamental tool for evaluating statements about population parameters based on the information observed in a sample. Generally speaking, any hypothesis test is based on the contrast between two opposing propositions: the null hypothesis, denoted by \(H_0\), and the alternative hypothesis, denoted by \(H_1\). In the most common case of a two-sided test, these hypotheses are expressed as: \[ \begin{cases} H_{0}: & \theta = \theta_0 \\ H_{1}: & \theta \neq \theta_0 \end{cases} \]
where \(\theta\) represents the population parameter of interest and \(\theta_0\) a reference value specified under the null hypothesis. Depending on the objective of the analysis, the alternative hypothesis can also be formulated unilaterally, for example, when you want to evaluate whether \(\theta > \theta_0\) or whether \(\theta < \theta_0\). The purpose of the test is to determine whether the evidence contained in the sample is strong enough to reject the null hypothesis in favor of the alternative hypothesis.
Suppose \(\bar{y}_1\) and \(\bar{y}_2\) represent the population means of a variable \(y\) in two different domains, for example, the mean income of male-headed households and the mean income of female-headed households. So, the parameter of interest then corresponds to the difference: \[ \Delta = \bar{y}_1 - \bar{y}_2 \]
Its sample estimator is defined as: \[ \hat{\Delta} = \hat{\bar{y}}_1 - \hat{\bar{y}}_2 \]
For which, its standard error is estimated by the following expression: \[ \widehat{se}(\hat{\Delta}) = \sqrt{ \widehat{Var}(\hat{\bar{y}}_1) + \widehat{Var}(\hat{\bar{y}}_2) - 2\widehat{Cov}(\hat{\bar{y}}_1, \hat{\bar{y}}_2) } \]
The presence of the covariance term is important because the estimates of both domains usually come from the same sample and, therefore, are not necessarily independent. Once the estimator and its standard error have been calculated, the hypothesis contrast is carried out using the following test statistic: \[ t = \frac{\hat{\Delta}} {\widehat{se}(\hat{\Delta})} \sim t_{df} \]
which approximately follows a Student \(t\) distribution with \((df)\) degrees of freedom. In practice, these degrees of freedom are usually approximated by \(df = n_I - H\), where \(n_I\) represents the number of primary sampling units and \(H\) the number of strata. Under the null hypothesis, high absolute values of the \(t\) statistic constitute evidence against \(H_0\). In a complementary manner, a confidence interval can also be constructed for the parameter \(\Delta\). For a confidence level of \((1-\alpha)100\%\), said interval is given by: \[ \hat{\Delta} \;\pm\; t_{(1-\alpha/2,df)} \, \hat{se}(\hat{\Delta}). \]
This interval provides a range of plausible values for the population difference and is a useful tool for evaluating both the magnitude and statistical significance of the observed differences between groups.
In R, these tests can be implemented using the svyttest() function of the survey package, which automatically incorporates the adjustments associated with the sample design. From the results presented in Table 3.22 it can be concluded that, with a confidence level of 95%, there is not sufficient statistical evidence to affirm that average incomes differ by sex.
| statistic | p_value | gl | ci_lower | ci_upper | estimated_difference |
|---|---|---|---|---|---|
| 1.36 | 0.176 | 118 | -12.8 | 69.4 | 28.3 |
Another hypothesis of interest consists of evaluating whether the average household income differs depending on the sex of the head within the urban area, as presented in Table 3.23. The results indicate that the null hypothesis \(H_0\) is not rejected, so there is not enough statistical evidence to affirm that average income differs by sex in the urban area.
urban_subset <- survey_design %>%
filter(Zone == "Urban")
ttest_urban <- svyttest(Income ~ Sex, design = urban_subset, level = 0.95)| statistic | p_value | gl | ci_lower | ci_upper | estimated_difference |
|---|---|---|---|---|---|
| 1.57 | 0.122 | 63 | -12.3 | 102 | 44.7 |
The procedure described is not limited to means, but can also be applied to proportions, totals, ratios or any function differentiable from totals. In all cases, the contrast is based on the point estimate, its variance (including covariances when applicable) and the comparison with the \(t\) distribution adjusted to the sample design. ## Contrast estimation {#estimacion-de-contrastes}
In the analysis of household surveys it is common that the interest is not limited to comparing only two populations, but several simultaneously. For example, it may be necessary to compare the average household income between different regions, geographic areas or population groups, in order to identify differences and establish patterns of inequality between them. In these types of situations, the mean difference tests presented above are limited, since they are designed to only compare pairs of populations.
To address more general comparisons, contrasts are used, which constitute a flexible tool to evaluate linear combinations of population parameters. In general terms, a contrast is defined as: \[ f\left(\theta_1,\theta_2,\ldots,\theta_J\right) = \sum_{j=1}^{J} a_j \theta_j, \]
where the coefficients \(a_j\) are known constants and \(\theta_j\) represent the parameters of interest. This approach allows a wide variety of comparisons between groups to be formulated and evaluated, including differences between means, multiple comparisons, and more complex combinations of parameters.
In R, methodological procedures for implementing contrasts in complex sampling designs are available through the svycontrast() function of the survey package. For example, if you want to compare the average income of two particular subpopulations (the Northern and Southern regions) you can use the contrast \(\bar{y}_{Norte} - \bar{y}_{Sur}\). Since the sample covers five regions in total, the contrast must be constructed by assigning coefficients only to the regions involved in the comparison and leaving coefficients equal to zero for the other regions. In this way, the contrast is defined as:
\[
1\times\hat{\bar{y}}_{Norte}+\left(-1\right)\times\hat{\bar{y}}_{Sur}+0\times\hat{\bar{y}}_{Centro}+0\times\hat{\bar{y}}_{Occidente}+0\times\hat{\bar{y}}_{Oriente}.
\]
This expression indicates that the contrast corresponds to the difference between the estimated means of the North and South regions, while the other regions do not participate in the comparison. In matrix form, the contrast can be written as: \[ \left[1,\,-1,\,0,\,0,\,0\right] \times \left[\begin{array}{c} \hat{\bar{y}}_{Norte}\\ \hat{\bar{y}}_{Sur}\\ \hat{\bar{y}}_{Centro}\\ \hat{\bar{y}}_{Occidente}\\ \hat{\bar{y}}_{Oriente} \end{array}\right]. \]
Consequently, the contrast vector associated with this comparison is \(\left[1,\,-1,\,0,\,0,\,0\right]\), where the positive and negative coefficients indicate the populations to be compared and the zeros represent the populations excluded from the contrast. The first step is to calculate the estimated means for each region using group_by() and survey_mean(), as presented in Table 3.24. The visible results object is constructed with srvyr; Additionally, an auxiliary object with a covariance matrix is created for subsequent contrasts. As a result, the estimated averages of income by region are obtained.
region_mean <- survey_design %>%
group_by(Region) %>%
summarise(mean = survey_mean(Income,
na.rm = TRUE,
vartype = c("se", "ci")))
region_mean_contrast <- svyby(formula = ~Income,
by = ~Region,
design = survey_design,
FUN = svymean,
na.rm = TRUE,
covmat = TRUE,
vartype = c("se", "ci"))| Region | mean | mean_se | mean_low | mean_upp |
|---|---|---|---|---|
| Norte | 552 | 55.4 | 443 | 662 |
| Sur | 626 | 62.4 | 502 | 749 |
| Centro | 651 | 61.5 | 529 | 772 |
| Occidente | 517 | 46.2 | 425 | 609 |
| Oriente | 542 | 71.7 | 400 | 684 |
The svycontrast() function returns the estimated contrast and its standard error. The arguments of this function are the averages of the estimated revenues (stat) and the contrast constants (contrasts), as shown in Table 3.25.
contrast_region_ns <- svycontrast(
stat = region_mean_contrast,
contrasts = list(north_south_difference = c(1, -1, 0, 0, 0))
)| contrast | estimate | variance | standard_error |
|---|---|---|---|
| north_south_difference | -73.4 | 6959 | 83.4 |
Note that these same results could be obtained by explicitly carrying out the process of constructing the contrast estimator and its estimated variance. If only the estimated average incomes of the North and South regions are considered, their difference is:
## [1] -73.4The next step is to calculate the variance and covariance matrix and from there extract the variances and covariances of the North and South regions, presented in Table 3.26:
| Region | Norte | Sur | Centro | Occidente | Oriente |
|---|---|---|---|---|---|
| Norte | 3065 | 0 | 0 | 0 | 0 |
| Sur | 0 | 3894 | 0 | 0 | 0 |
| Centro | 0 | 0 | 3778 | 0 | 0 |
| Occidente | 0 | 0 | 0 | 2136 | 0 |
| Oriente | 0 | 0 | 0 | 0 | 5136 |
To calculate the standard error of the difference (contrast), the properties of the variance will be used, as follows: \[ \widehat{se}\left(\hat{\bar{y}}_{Norte}-\hat{\bar{y}}_{Sur}\right)=\sqrt{\widehat {Var}\left(\hat{\bar{y}}_{Norte}\right)+ \widehat{Var}\left(\hat{\bar{y}}_{Sur}\right)- 2\, \widehat{Cov}\left(\hat{\bar{y}}_{Norte},\hat{\bar{y}}_{Sur}\right)} \]
Therefore:
## [1] 78.2Which corresponds to the same result obtained previously with the svycontrast() function. In addition to comparing only two populations, contrasts allow several comparisons of interest to be evaluated simultaneously between different groups.
On the other hand, suppose you want to compare average incomes between different geographic regions. In particular, contrasts such as \(\bar{y}_{Norte} - \bar{y}_{Centro}\), \(\bar{y}_{Sur} - \bar{y}_{Centro}\) and \(\bar{y}_{Occidente} - \bar{y}_{Oriente}\) could be considered. Each of these expressions represents a specific linear contrast between regional means. Jointly, these contrasts can be organized using the following contrast matrix: \[ \left[\begin{array}{ccccc} 1 & 0 & -1 & 0 & 0\\ 0 & 1 & -1 & 0 & 0\\ 0 & 0 & 0 & 1 & -1 \end{array}\right] \]
In this matrix, each row defines a different contrast and each column corresponds to one of the regions considered in the analysis. The positive and negative coefficients indicate the means that participate in each comparison, while the zeros represent the regions that do not participate in the corresponding contrast. Below is the implementation of the contrasts in R, the results of which are presented in Table 3.27. From these results, it can be concluded that the Southern and Central regions have the most similar average household incomes, given that the estimated difference between both regions is the smallest among the contrasts evaluated.
contrast_regions <- svycontrast(
stat = region_mean_contrast,
contrasts = list(
north_south = c(1, 0, -1, 0, 0),
south_center = c(0, 1, -1, 0, 0),
west_east = c(0, 0, 0, 1, -1)
)
)| contrast | estimate | variance | standard_error |
|---|---|---|---|
| north_south | -98.4 | 6843 | 82.7 |
| south_center | -25.0 | 7673 | 87.6 |
| west_east | -24.7 | 7272 | 85.3 |
It is also possible to build contrasts between variables associated with different population groups, as occurs when comparing average income according to sex. Following the same procedure as the previous example, the first step is to estimate the average income for each group, in this case men and women, as presented in Table 3.28.
sex_mean <- survey_design %>%
group_by(Sex) %>%
summarise(mean = survey_mean(Income,
na.rm = TRUE,
vartype = c("se", "ci")))
sex_mean_contrast <- svyby(formula = ~Income,
by = ~Sex,
design = survey_design,
FUN = svymean,
na.rm = TRUE,
covmat = TRUE,
vartype = c("se", "ci"))| Sex | mean | mean_se | mean_low | mean_upp |
|---|---|---|---|---|
| Female | 558 | 25.8 | 506 | 609 |
| Male | 586 | 34.6 | 517 | 654 |
In this case, the contrast of interest is given by \(\bar{y}_{F} - \bar{y}_{M}\), which allows quantifying the difference between the average income of women and that of men. Next, using the function svycontrast() of the survey package, the contrast estimation is obtained, the results of which are presented in Table 3.29. From these results, it is concluded that, on average, men receive 28.3 monetary units more than women, with an estimated standard error of 19.8 monetary units.
contrast_sex <- svycontrast(
stat = sex_mean_contrast,
contrasts = list(sex_difference = c(1, -1))
)| contrast | estimate | variance | standard_error |
|---|---|---|---|
| sex_difference | -28.3 | 431 | 20.8 |
Since the contrasts correspond to linear functions of parameters, it is also possible to apply them to ratio estimators. An example of this is to compare the relationship between expenses and income according to sex. In this case, first the expense-income ratio is estimated for each group and subsequently the contrast between these ratios is constructed. Below are the computational codes used to perform this analysis, the results of which are shown in Table 3.30:
sex_ratio <- survey_design %>%
group_by(Sex) %>%
summarise(ratio = survey_ratio(Income,
Expenditure,
na.rm = TRUE,
vartype = c("se", "ci")))
sex_ratio_contrast <- svyby(formula = ~Income,
by = ~Sex,
denominator = ~Expenditure,
design = survey_design,
FUN = svyratio,
na.rm = TRUE,
covmat = TRUE,
vartype = c("se", "ci"))| Sex | ratio | ratio_se | ratio_low | ratio_upp |
|---|---|---|---|---|
| Female | 1.52 | 0.046 | 1.43 | 1.61 |
| Male | 1.56 | 0.070 | 1.43 | 1.70 |
The svycontrast() function is not limited only to the comparison of means or totals, but also allows the construction of contrasts between other parameters of interest, such as ratios and proportions estimated from complex sampling designs. The computational codes used to perform this analysis are presented below, the results of which are shown in Table 3.31. From these results, it can be concluded that the difference between the estimated ratios is 0.045 in favor of men.
contrast_sex_ratio <- svycontrast(
stat = sex_ratio_contrast,
contrasts = list(sex_difference = c(1, -1))
)| contrast | estimate | standard_error | variance |
|---|---|---|---|
| sex_difference | -0.046 | 0.042 | 0.002 |