与intercept=TRUE相比，Glmnet与intercept=FALSE不同，对于x中的拦截而言，Glmnet与penalty.factor=0不同。

Question

我对glmnet和penalty.factor的选择都是新手。vignette说它“对于某些变量可以是0，这意味着没有收缩，并且这个变量总是包含在模型中。”较长的PDF文档有代码。因此，我预计，使用intercept = TRUE运行回归，在x中不运行常量，将与intercept = FALSE一样，在x中使用penalty.factor = 0时运行常量。但下面的代码表明并非如此:后一种情况的截距为0，另外两个系数比前者大20%。

library("glmnet")

set.seed(7)

# penalty for the intercept
intercept_penalty <- 0

# Simulate data with 2 features
num_regressors <- 2
num_observations <- 100
X <- matrix(rnorm(num_regressors * num_observations),
            ncol = num_regressors,
            nrow = num_observations)

# Add an intercept in the right-hand side matrix: X1 = (intercept + X)
X1 <- cbind(matrix(1, ncol = 1, nrow = num_observations), X)

# Set random parameters for the features
beta <- runif(1 + num_regressors)

# Generate observations for the left-hand side
Y <- X1 %*% beta + rnorm(num_observations) / 10

# run OLS
ols <- lm(Y ~ X)
coef_ols <- coef(ols)

# Run glmnet with an intercept in the command, not in the matrix
fit <- glmnet(y = Y,
              x = X,
              intercept = T,
              penalty.factor = rep(1, num_regressors),
              lambda = 0)
coef_intercept_equal_true <- coef(fit)

# run glmnet with an intercept in the matrix with a penalty
# factor of intercept_penalty for the intercept and 1 for the rest
fit_intercept_equal_false <- glmnet(y = Y,
              x = X1,
              intercept = F,
              penalty.factor = c(intercept_penalty, rep(1, num_regressors)),
              lambda = 0)
coef_intercept_equal_false <- coef(fit_intercept_equal_false)

# Compare all three methods in a data frame
# For lasso_intercept_equal_false, the index starts at 2 because
# position 1 is reserved for intercepts, which is missing in this case
comparison <- data.frame(original = beta,
                         ols = coef_ols,
                         lasso_intercept_equal_true = coef_intercept_equal_true[1:length(coef_intercept_equal_true)],
                         lasso_intercept_equal_false = coef_intercept_equal_false[2:length(coef_intercept_equal_false)]
)

comparison$difference <- comparison$lasso_intercept_equal_false - comparison$lasso_intercept_equal_true
comparison

此外，对于截距项，无论intercept_penalty是否等于0、1、3000、-10等，对于不同的惩罚因子，这种差异都是相同的。这种差异与正罚，例如lambda = 0.01相似。

如果这不是一个bug，penalty.factor的正确用法是什么？

Answer 1

我联系了作者，他确认这是一个bug，并补充说，这是他的错误修复列表。在此期间，解决方法是将回归者作为中心，例如

fit_centered <- glmnet(y = Y,
                       x = scale(X1, T, F),
                       intercept = F,
                       lambda = 0)

在这种情况下，惩罚因素并不重要。下面是一个修改后的脚本，比较OLS、LASSO与拦截、LASSO无拦截和LASSO与居中回归者的比较：

library("glmnet")

set.seed(7)

# Simulate data with 2 features
num_regressors <- 2
num_observations <- 100
X <- matrix(rnorm(num_regressors * num_observations),
            ncol = num_regressors,
            nrow = num_observations)

# Add an intercept in the right-hand side matrix: X1 = (intercept + X)
X1 <- cbind(matrix(1, ncol = 1, nrow = num_observations), X)

# Set random parameters for the features
beta <- runif(1 + num_regressors)

# Generate observations for the left-hand side
Y <- X1 %*% beta + rnorm(num_observations) / 10

# run OLS
ols <- lm(Y ~ X)
coef_ols <- coef(ols)

# Run glmnet with an intercept in the command, not in the matrix
fit <- glmnet(y = Y,
              x = X,
              intercept = T,
              penalty.factor = rep(1, num_regressors),
              lambda = 0)
coef_intercept <- coef(fit)

# run glmnet with an intercept in the matrix with a penalty
# factor of 0 for the intercept and 1 for the rest
fit_no_intercept <- glmnet(y = Y,
              x = X1,
              intercept = F,
              lambda = 0)
coef_no_intercept <- coef(fit_no_intercept)

# run glmnet with an intercept in the matrix with a penalty
# factor of 0 for the intercept and 1 for the rest
# If x is centered, it works (even though y is not centered). Center it with:
#   X1 - matrix(colMeans(X1), nrow = num_observations, ncol = 1 + num_regressors, byrow = T)
# or with
# X1_centered = scale(X1, T, F)

fit_centered <- glmnet(y = Y,
                      x = scale(X1, T, F),
                      intercept = F,
                      lambda = 0)
coef_centered <- coef(fit_centered)

# Compare all three methods in a data frame
# For lasso_intercept and the others, the index starts at 2 because
# position 1 is reserved for intercepts, which is missing in this case
comparison <- data.frame(ols = coef_ols,
                         lasso_intercept = coef_intercept[1:length(coef_intercept)],
                         lasso_no_intercept = coef_no_intercept[2:length(coef_no_intercept)],
                         lasso_centered = coef_centered[2:length(coef_centered)]
)


comparison$diff_intercept <- comparison$lasso_intercept - comparison$lasso_no_intercept
comparison$diff_centered <- comparison$lasso_centered - comparison$lasso_intercept
comparison

答案是：

                  ols lasso_intercept lasso_no_intercept lasso_centered diff_intercept diff_centered
(Intercept) 0.9748302       0.9748302          0.0000000      0.0000000      0.9748302 -9.748302e-01
X1          0.6559541       0.6559541          0.7974851      0.6559541     -0.1415309  2.220446e-16
X2          0.7986957       0.7986957          0.9344306      0.7986957     -0.1357348  4.440892e-16

对于具有中心回归元的套索，估计截距为0，但其他系数与带截距的拉索相同。