Automatic Relevance Determination (ARD) Regression System

An Automatic Relevance Determination (ARD) Regression System is a Bayesian Regression System that implements an ARD Regression Algorithm to solve a ARD Regression Task.

• AKA: Sparse Bayesian Learning System, Relevance Vector Machine System, Bayesian ARD Regression System.
  • …
• Example(s):
  • sklearn.linear_model.ARDRegression [1]:
    • Automatic Relevance Determination Regression (ARD)
• Counter-Example(s):
  • Bayesian Ridge Regression System.
  • LASSO Regression System.
  • Ridge Regression System.
  • ElasticNet System.
• See: Cross-Validation Task, L1-Norm, L2-norm, Bayesian Inference.

---

---

References

2017

• (Scikit Learn, 2017) ⇒ http://scikit-learn.org/stable/modules/linear_model.html#automatic-relevance-determination-ard Retrieved:2017-09-17
  • QUOTE: ARDRegression is very similar to Bayesian Ridge Regression, but can lead to sparser weights w . ARDRegression poses a different prior over w, by dropping the assumption of the Gaussian being spherical.

    Instead, the distribution over w is assumed to be an axis-parallel, elliptical Gaussian distribution.

    This means each weight [math]\displaystyle{ w_{i} }[/math] is drawn from a Gaussian distribution, centered on zero and with a precision [math]\displaystyle{ \lambda_{i} }[/math]:

    [math]\displaystyle{ p(w|\lambda) = \mathcal{N}(w|0,A^{-1}) }[/math]

    with [math]\displaystyle{ diag \; (A) = \lambda = \{\lambda_{1},...,\lambda_{p}\} }[/math].

    In contrast to Bayesian Ridge Regression, each coordinate of [math]\displaystyle{ w_{i} }[/math] has its own standard deviation [math]\displaystyle{ \lambda_i }[/math]. The prior over all [math]\displaystyle{ \lambda_i }[/math] is chosen to be the same gamma distribution given by hyperparameters [math]\displaystyle{ \lambda_1 }[/math] and [math]\displaystyle{ \lambda_2 }[/math]. ARD is also known in the literature as Sparse Bayesian Learning and Relevance Vector Machine .

---