Linear Generative Classification Algorithm

A Linear Generative Classification Algorithm is a generative classification algorithm that uses a Linear Classification Model.

• Example(s):
  • Linear Discriminant Analysis.
  • Naive-Bayes Classifier.
  • …
• Counter-Example(s):
  • a Linear Discriminative Classification Algorithm, such as linear logistic regression.
  • a Quadratic Generative Classification Algorithm.
• See: Generative Statistical Model ..

---

---

References

2011

• http://en.wikipedia.org/wiki/Linear_classifier#Generative_models_vs._discriminative_models
  • There are two broad classes of methods for determining the parameters of a linear classifier [math]\displaystyle{ \vec w }[/math]. Methods of the first class model conditional density functions [math]\displaystyle{ P(\vec x|{\rm class}) }[/math]. Examples of such algorithms include:
    • Linear Discriminant Analysis (or Fisher's linear discriminant) (LDA) — assumes Gaussian conditional density models.
    • Naive Bayes classifier — assumes independent binomial conditional density models.

2004

• (Bouchard & Triggs, 2004) ⇒ Guillaume Bouchard, and Bill Triggs. (2004). “The Trade-off Between Generative and Discriminative Classifiers.” In: Proceedings of COMPSTAT 2004.
  • QUOTE: Well known generative-discriminative pairs include Linear Discriminant Analysis (LDA) vs. Linear logistic regression and naive Bayes vs. Generalized Additive Models (GAM). Many authors have already studied these models e.g. [5,6]. Under the assumption that the underlying distributions are Gaussian with equal covariances, it is known that LDA requires less data than its discriminative counterpart, linear logistic regression [3]. More generally, it is known that generative classifiers have a smaller variance than. … Conversely, the generative approach converges to the best model for the joint distribution [math]\displaystyle{ p(x,y) }[/math] but the resulting conditional density is usually a biased classifier unless its [math]\displaystyle{ p_\theta(x) }[/math] part is an accurate model for [math]\displaystyle{ p(x) }[/math]. In real world problems the assumed generative model is rarely exact, and asymptotically, a discriminative classifier should typically be preferred [9,5].

---