Contribution Constrained Regularization

Harry He

2025-06-26

Contribution Cap Regularization

Consider a linear regression problem with elastic net regularization. THe optimization objective is: \[\arg\min_{\beta}||Y-X\beta||_2^2+\lambda\Big(\alpha||\beta||_1+(1-\alpha)||\beta||_2^2\Big)\]

In many practical applications, it may be desirable to impose additional structure on the model coefficients. For instance, one might want to enforce monotonicity, apply budget constraints, or–our focus here–limit the contribution of each covariates to the final prediction. This is especially relevant in settings where:

We define the contribution of covariates \(i\) as the absolute value of the product \(\beta_iZ_i\), where \(Z_i\) is a data matrix used to evaluate feature contributions, and seek to penalize contributions that exceed a specific upper bound \(L_i\). This leads to either a hard-constrained formulation: \[\arg\min_{\beta}||Y-X\beta||_2^2+\lambda\Big(\alpha||\beta||_1+(1-\alpha)||\beta||_2^2\Big)\quad\text{s.t.}\quad \beta_iZ_i\leq L\quad\forall i\] or a soft-constrained formulation, where the violation of contribution caps is penalized: \[\arg\min_{\beta}||Y-X\beta||_2^2+\lambda\Big(\alpha||\beta||_1+(1-\alpha)||\beta||_2^2\Big)+\mu\sum_{i=1}^k\Big|\Big|\max\big(0,|\beta_iZ_i|-L\big)\Big|\Big|_p^p\] where \(k\) is the number of covariates and \(L\) is the contribution cap vector. When \(p=2\), the cap penalty becomes a sum of squared violations, which is differentiable almost everywhere and easier to optimize compared to an \(\mathcal{l}_1\)-based formulation. While the overall penalty remains non-smooth due to the \(\max(0,\cdot)\), the use of squared penalty reduces sensitivity near the boundary and facilitates convergence with quasi-Newton methods.

This formulation resembles generalized LASSO-type problems (Tibshirani and Taylor 2011) and constrained penalization frameworks (James, Paulson, and Rusmevichientong 2020), but with an important difference: the penalty is applied to the feature contributions, not the coefficients themselves. Because the capped contributions involve data-dependent terms (\(\beta_iX_i\)) and a non-linear transformation, there is no simple matrix \(D\) such that this penalty reduces to a standard form \(|\|D\beta\||_1\).

Optimization Strategy

The inclusion of the contribution cap penalty renders off-the-shelf solvers like glmnet unsuitable. Instead, we turn to general-purpose optimizers:

To align with glmnet conventions, we rewrite the loss in a normalized form as \[\min_{\beta_0,\beta}\frac{1}{2N}\sum_{i=1}^N\Big(y_i-(\beta_0+x_i^T\beta)\Big)^2+\lambda\bigg[\frac{(1-\alpha)}{2}||\beta||_2^2+\alpha||\beta||_1\bigg]+\mu\sum_{i=1}^k\max\big(0,|\beta_iZ_i|-L\big)^2\]

Basic Usage of Capnet

To see how capnet works in practice. We first call the data simulator, which generates the X and y matrices, a true beta vector, and a sample contribution cap limit. The default setting produces a input matrix of size 100 \(\times\) 20.

example <- simulate_capnet_data()
X <- example$X
y <- example$y
n <- nrow(X)
p <- ncol(X)
L <- example$L
beta_init <- rep(0, p + 1)

We then call capnet.cv to find the best \(\alpha\) and \(\lambda\) values through cross-validation. The default method uses 5-fold cross-validation but more complicated split methods can be specified.

capnet.cv1 <- cv_capnet(X, y, 0, L, standardize = FALSE)
alpha1 <- capnet.cv1$best_alpha
lambda1 <- capnet.cv1$best_lambda
capnet.mod1 <- capnet(X, y, alpha = alpha1, lambda = lambda1,
                 mu = 0, L = L, standardize = FALSE)

Best alpha value: 1.000 | Best lambda value 0.0179

capnet.cv2 <- cv_capnet(X, y, 1, L, standardize = FALSE)
alpha2 <- capnet.cv2$best_alpha
lambda2 <- capnet.cv2$best_lambda
capnet.mod2 <- capnet(X, y, alpha = alpha2, lambda = lambda2,
                 mu = 0.5, L = L, standardize = FALSE)

Best alpha value: 1.000 | Best lambda value 0.0067

plots <- plot_redistribution(capnet.mod1, capnet.mod2, plot = "raw")
plots$beta

plots$contribution

## [1] "System runtime for unconstrained 5-fold cross-validation"
##    user  system elapsed 
##  14.304   0.508  14.826
## [1] "System runtime for constrained 5-fold cross-validation"
##    user  system elapsed 
##  15.303   0.513  15.829
## [1] "System runtime for constrained model fit"
##    user  system elapsed 
##   0.023   0.000   0.023

Additional Parameters

Standardization

Similar to glmnet, capnet also has a standardize parameter that is default to TRUE. This controls standardization of the X and y matrices. It is often recommended to scale X and y terms for elastic net regressions and that also applies to capnet as well. The function has been optimized to handle variable scaling such that the penalty term, which is calculated at the original variable scale, is scaled to be in the same unit as the rest of the loss and gradient terms to avoid convergence failure due to unit mismatch. Therefore, the user should supply newx and L in the same scale as X and y regardless of the value of the standardize parameter.

Multiplier

The multiplier parameter in capnet controls the scaling applied to feature contributions. It is particularly useful when the contribution cap needs to be evaluated across multiple observations with non-uniform scaling factors for each observation. For example, if the user has a constant contribution limit L but wishes to evaluate contributions for several prediction points, each requiring different scaling, multiplier can be specified as a \(n\times1\) vector of scalars corresponding to each observation. This avoids the need to manually transform L into a matrix aligned with each observation. Currently, the implementation does not support L as a matrix input, so multiplier provides a flexible alternative for handling observation-specific scaling requirements efficiently.

Box Constraint

The capnet function also allows users to enforce box constraints on model parameters using the upper.limits and lower.limits arguments. These specify the maximum and minimum allowable values for each coefficient, respectively. This is useful when prior knowledge or practical considerations dictate that parameter estimates must remain within specific bounds. The box constraints is achieved via gradient masking, and tol sets a tolerance threshold for enforcing such box constraints. If a parameter is within tol of lower or upper bounds and the current gradient would push it toward the limit, the gradient is manually set to zero for that parameter. The users can thus increase the tol variable if default value fails to enforce the box constraint and decrease the tol if it restrict parameter updates prematurally.

Parameter initialization: Because box constraints are achieved via gradient masking, the initial parameter values par needs to be within the specified limits. If initial values fall outside these bounds, optimization may fail to converge or produce invalid results. By default, par initializes to zeros, so users should adjust the initialization if zeros fall outside the desired parameter range.

Convergence Failure

Because capnet relies on the lbfgs package for optimization, convergence failure can be caused by multiple factors and may be hard for users to detect and debug. The check.finite parameter can help the user determine whether convergence failure occurred due to non-finite values when calculating loss and gradients. By setting check.finite=TRUE, the function will catch and throw non-finite value errors. It is also important to note that the model does not stop or return error even when convergence failed, so users are advised to manually check model convergence after fitting. Because lbfgs does not handle and store messages for all convergence failure codes, the message output may not provide useful information on why convergence failed. Generally, when users encounter a \(-998\) error code, it is usually that the line search algorithm failed because the gradient values are too unstable.

Use cases for Contribution Cap Penalty

Finance

Social Science

Machine Learning and Statistical Modeling

Healthcare and Natural Science

James, Gareth M., Courtney Paulson, and Paat Rusmevichientong. 2020. “Penalized and Constrained Optimization: An Application to High-Dimensional Website Advertising.” Journal of American Statistical Association 115 (529): 107–22.
Tibshirani, Ryan J., and Jonathan Taylor. 2011. “The Solution Path of the Generalized LASSO.” THe Annals of Statistics 39 (3): 1335–71.