Closed-Form Linear Regression Algorithm

A Closed-Form Linear Regression Algorithm is a linear regression algorithm that is a closed-form algorithm.

• See: Approximate Linear Regression Algorithm.

---

---

References

2012

• https://stats.stackexchange.com/questions/23128/solving-for-regression-parameters-in-closed-form-vs-gradient-descent
  • QUOTE: What is the advantage of using an iterative algorithm like gradient descent over a closed-form solution in general, when one is available? …
    … Now, imagine that X is a very large but sparse matrix. e.g. X might have 100,000 columns and 1,000,000 rows, but only 0.001% of the entries in X are nonzero. There are specialized data structures for storing only the nonzero entries of such sparse matrices.

    Also imagine that we're unlucky, and XTX is a fairly dense matrix with a much higher percentage of nonzero entries. Storing a dense 100,000 by 100,000 element XTX matrix would then require 1×1010 floating point numbers (at 8 bytes per number, this comes to 80 gigabytes.) This would be impractical to store on anything but a supercomputer. Furthermore, the inverse of this matrix (or more commonly a Cholesky factor) would also tend to have mostly nonzero entries.

    However, there are iterative methods for solving the least squares problem that require no more storage than X, y, and β̂ and never explicitly form the matrix product XTX.

    In this situation, using an iterative method is much more computationally efficient than using the closed form solution to the least squares problem.

---