Regularization Penalty Parameter

A Regularization Penalty Parameter is a tuning parameter (used in a regularized training algorithm) that penalizes divergence from the mean.

• AKA: Regularisation Coefficient.
• Context:
  • It can help to overcome overfitting.
  • It can range from an L1 Regularized Parameter to being an L2 Regularized Parameter.
• See: Regularized Algorithm, Tuning Parameter, Penalized MLE, MAP Estimator.

---

---

References

2011

• (Chang & Lin, 2011) ⇒ Chih-Chung Chang, and Chih-Jen Lin. (2011). “LIBSVM: A Library for Support Vector Machines.” In: ACM Transactions on Intelligent Systems and Technology (TIST) Journal, 2(3). doi:10.1145/1961189.1961199
  • QUOTE: ... where [math]\displaystyle{ \phi(x_i) }[/math] maps [math]\displaystyle{ x_i }[/math] into a higher-dimensional space and C > 0 is the regularization parameter. Due to the possible high dimensionality of the vector variable [math]\displaystyle{ w }[/math], usually we solve the following dual problem. ...

2010

• (Jin et al., 2010) ⇒ Xiao-Bo Jin, Cheng-Lin Liu, and Xinwen Hou. (2010). “Regularized Margin-based Conditional Log-likelihood Loss for Prototype Learning.” In: Pattern Recognition Journal, 43(7).In: Pattern Recognition Journal, 43(7). doi:10.1016/j.patcog.2010.01.013

2008

• (Zou & Li, 2008) ⇒ Hui Zou, and Runze Li. (2008). “One-Step Sparse Estimates in Nonconcave Penalized Likelihood Models (with discussion).” In: Annals of Statistics. 36.
  • QUOTE: Statistically, we show that if the regularization parameter is appropriately chosen, the one-step LLA estimates enjoy the oracle properties with good initial estimators.

2004

• (Hastie et al., 2004) ⇒ Trevor Hastie, Saharon Rosset, Robert Tibshirani, and Ji Zhu. (2004). “The Entire Regularization Path for the Support Vector Machine.” In: The Journal of Machine Learning Research, 5.
  • QUOTE: ... The user has to supply values for the tuning parameters: the regularization cost parameter, and the kernel parameters. It seems a common practice is to use a default value for the cost parameter, ...

    It seems that the regularization parameter C (or l) is often regarded as a genuine “nuisance” in the community of SVM users. Software packages, such as the widely used SVMlight (Joachims, 1999), provide default settings for C, which are then used without much further exploration. A recent introductory document (Hsu et al., 2003) supporting the LIBSVM package does encourage grid search for C.

---