Bayesian Regression System

A Bayesian Regression System is a Regression System that implements an Bayesian Regression Algorithm to solve a Bayesian Regression Task.

• AKA: Bayesian Regressor.
• Context:
  • It ranges from being a ARD Regression System to being a Bayesian Ridge Regression System.
  • It ranges from being a Linear Bayesian Regression System to being a Nonlinear Bayesian Regression System.
  • …
• Example(s):
  • sklearn.linear_model.ARDRegression [1].
  • sklearn.linear_model.BayesianRidge [2].
  • …
• Counter-Example(s):
  • LASSO Regression System.
  • Ridge Regression System.
  • ElasticNet System.
  • Robustness Regression System.
• See: Cross-Validation Task, Regression Analysis Task, Bayesian Inference, Bayes Theorem.

---

---

References

2017

• (Scikit Learn, 2017) ⇒ http://scikit-learn.org/stable/modules/linear_model.html#bayesian-regression Retrieved:2017-09-17
  • QUOTE: Bayesian regression techniques can be used to include regularization parameters in the estimation procedure: the regularization parameter is not set in a hard sense but tuned to the data at hand.

    This can be done by introducing uninformative priors over the hyperparameters of the model. The [math]\displaystyle{ \ell_{2} }[/math] regularization used in Ridge Regression is equivalent to finding a maximum a posteriori estimation under a Gaussian prior over the parameters w with precision [math]\displaystyle{ \lambda^{-1} }[/math]. Instead of setting lambda manually, it is possible to treat it as a random variable to be estimated from the data.

    To obtain a fully probabilistic model, the output [math]\displaystyle{ y }[/math] is assumed to be Gaussian distributed around [math]\displaystyle{ X w }[/math]:

    [math]\displaystyle{ p(y|X,w,\alpha) = \mathcal{N}(y|X w,\alpha) }[/math]

    Alpha is again treated as a random variable that is to be estimated from the data.

    The advantages of Bayesian Regression are:

    • It adapts to the data at hand.
    • It can be used to include regularization parameters in the estimation procedure.

The disadvantages of Bayesian regression include:

• Inference of the model can be time consuming.

---