Locally Weighted Regression System

A Locally Weighted Regression System is an nonparametric regression system which implements a Locally Weighted Regression Algorithm to solve a Locally Weighted Regression Task.

• AKA: LWR System.
• Example(s):
  • statsmodels.nonparametric.smoothers_lowess.lowess[1]
  • lowess.py [2][3] [4]
  • lwpr.c[5]
• Counter-Example(s):
  • sklearn.neighbors.RadiusNeighborsRegressor[6]
  • Gaussian Process Regression System.
  • Ridge Regression System.
  • Kernel Regression System.
• See: Nonparametric Regression Task, Lazy Learning Task, Distance Metric, Weighting Function, Distance Function, k-Nearest Neighbor (kNN) Algorithm.

---

---

References

2017a

• (StatsModels, 2017) ⇒ http://www.statsmodels.org/devel/generated/statsmodels.nonparametric.smoothers_lowess.lowess.html Retrieved: 2017-09-10
  • QUOTE: This lowess function implements the algorithm given in the reference below using local linear estimates.

    Suppose the input data has N points. The algorithm works by estimating the smooth y_i by taking the frac*N closest points to (x_i,y_i) based on their x values and estimating y_i using a weighted linear regression. The weight for (x_j,y_j) is tricube function applied to abs(x_i-x_j).

    If it > 1, then further weighted local linear regressions are performed, where the weights are the same as above times the _lowess_bisquare function of the residuals. Each iteration takes approximately the same amount of time as the original fit, so these iterations are expensive. They are most useful when the noise has extremely heavy tails, such as Cauchy noise. Noise with less heavy-tails, such as t-distributions with df>2, are less problematic. The weights downgrade the influence of points with large residuals. In the extreme case, points whose residuals are larger than 6 times the median absolute residual are given weight 0.

    delta can be used to save computations. For each x_i, regressions are skipped for points closer than delta. The next regression is fit for the farthest point within delta of x_i and all points in between are estimated by linearly interpolating between the two regression fits.

    Judicious choice of delta can cut computation time considerably for large data (N > 5000). A good choice is delta = 0.01 * range(exog).

    Some experimentation is likely required to find a good choice of frac and iter for a particular dataset.

2017b

• (Gramfort, 2017) ⇒ Alexandre Gramfort (2017). “LOWESS : Locally weighted regression" https://gist.github.com/agramfort/850437 Retrieved: 2017-09-10
  • QUOTE: This module implements the Lowess function for nonparametric regression.

    Functions: lowess Fit a smooth nonparametric regression curve to a scatterplot.

    For more information, see William S. Cleveland: "Robust locally weighted regression and smoothing scatterplots", Journal of the American Statistical Association, December 1979, volume 74, number 368, pp. 829-836.

    William S. Cleveland and Susan J. Devlin: "Locally weighted regression: An approach to regression analysis by local fitting", Journal of the American Statistical Association, September 1988, volume 83, number 403, pp. 596-610.

    Authors: Alexandre Gramfort <alexandre.gramfort@telecom-paristech.fr>

    License: BSD (3-clause)

2017c

• (cs.cmu.edu, 2017) ⇒ http://www.cs.cmu.edu/afs/cs.cmu.edu/project/learn-43/lib/photoz/.g/web/lwpr_src/lwpr.html Retrieved: 2017-09-10
  • QUOTE: lwpr.c is an unsophisticated, stand-alone implementation of locally weighted linear regression. It can be compiled with the following command: gcc lwpr.c -lm -o lwpr. To run the program, execute the command:

    lwpr <datafile> <testfile> <kwidth>

    where datafile contains the training set, testfile contains the test set, and kwidth is the kernel width. Each row of the datafile should contain exactly the number of input dimensions + 1 numbers, separated by spaces. The testfile should have the same format. It is important not to place any blank lines at the beginning of the files, or any stray characters throughout the files.

    The program will use points in the datafile to predict the output for each point in the testfile and print the predicted and actual outputs. It will also print the root mean square error of the predicted outputs. The training and test sets used in the examples above are gauss.train, gauss.test , sin.train, and sin.test.

    lwpr.c can be easily modified to implement higher order polynomials (or any other set of basis functions, for that matter). Because the code does not scale the input variables, the proper kernel width may be several orders of magnitude larger or smaller than the ones shown in the examples above. If you find that many of the predicted outputs are near 0 when they should not be, this generally indicates that the kernel width is far too small.

---