How important is that variable?

When modeling any phenomena by including explanatory variables that highly relates the variable of interest, one question arises: which of the auxiliary variables have a higher influence on the response? I am not writing about significance testing or something like this. I am just thinking like a researcher who wants to know the ranking of variables that influence the response and their related weight.

There are a variety of methods that try to answer that question. The one inducing this thread is very simple: isolate units from variables. Assume a linear model with the following structure (for the sake of simplicity, assume only two explanatory variables):

\[ y = \beta_1 x_1 + \beta_2 x_2 + \varepsilon \]

If you assume this model as true and \(\beta_i > 0\), then the influence of variable \(x_i\), over response \(y\) could be found when isolating measure units from variables. Then, one could fit a model over the standardized variables (explanatory and response) and then directly comparing the regression coefficients. Another way to do this is by means of the following expression:

\[ I(i) = \frac{\beta_i}{sd(\beta_i)} = \beta_i\frac{ sd(x_i)}{sd(y)}\]

For example, let’s consider the following model \(y = -500 x_1 + 50 x_2 + \varepsilon\), then the relative importance of the first and second variable is around 500/(500+50) = 0.9, and 50/(500+50) = 0.1, respectively. The following code shows how to perform this simple analysis in R.

n <- 10000

x1 <- runif(n)
x2 <- runif(n)
y <- -500 * x1 + 50 * x2 + rnorm(n)

model <- lm(y ~ 0 + x1 + x2)

# 1a. Standardized betas
summary(model)$coe[,2]

        x1         x2 
0.02588824 0.02566665 

sd.betas <- summary(model)$coe[,2]
betas <- model$coefficients
imp <- abs(betas)/sd.betas
imp <- imp/sum(imp)
imp

        x1         x2 
0.90833992 0.09166008 

# 1b. Standardized betas
imp1 <- abs(model$coefficients[1] * sd(x1)/sd(y))
imp2 <- abs(model$coefficients[2] * sd(x2)/sd(y))

imp1 / (imp1 + imp2)

       x1 
0.9085622 

imp2 / (imp1 + imp2)

        x2 
0.09143784 

# 2. Standardized variables
model2 <- lm(I(scale(y)) ~ 0 + I(scale(x1)) + I(scale(x2)))
summary(model2)

Call:
lm(formula = I(scale(y)) ~ 0 + I(scale(x1)) + I(scale(x2)))

Residuals:
       Min         1Q     Median         3Q        Max 
-0.0264801 -0.0045735  0.0000143  0.0046105  0.0251890 

Coefficients:
               Estimate Std. Error t value Pr(>|t|)    
I(scale(x1)) -9.943e-01  6.816e-05  -14589   <2e-16 ***
I(scale(x2))  1.001e-01  6.816e-05    1468   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.006815 on 9998 degrees of freedom
Multiple R-squared:      1, Adjusted R-squared:      1 
F-statistic: 1.076e+08 on 2 and 9998 DF,  p-value: < 2.2e-16

abs(model2$coefficients)/sum(abs(model2$coefficients))

I(scale(x1)) I(scale(x2)) 
  0.90856864   0.09143136