Deliberately trying to overfit a model

wcraine

We are working in Notebook 4, using sample_dat, predicting default_rate from SAT_avg using polynomial functions. I was trying to make the point to my class that 2 points determine a line, 3 points determine a parabola, etc., so we could hit each of our 20 points with a degree 19 polynomial and get R² = 100%.

R wouldn't do it, and I realized that 2 of the points in sample_dat had the same SAT_avg. So, it's impossible to hit all the points exactly with a polynomial function. Cool.

But, then I started knocking down the degree of the function until R could make one. Degree 18, 17, 16, 15, and 14 were all no-go. It could make degree 13.

The error was:
Error in poly(SAT_avg, 14): 'degree' must be less than number of unique points

Why couldn't it make the others? This error doesn't even really describe why it couldn't make degree 19. There were 20 unique points, just not 20 points without equal SAT_avg.

I don't understand enough about what R is doing behind the scenes to figure this out.

Thanks,
Bill

dylarm

You are technically correct about the polynomial degree and overfitting. Unfortunately though, once you start getting into higher and higher degrees, physical/software limitations start coming into play as well.

The short answer is that due to the way that poly is implemented in R, large exponents result in numerical overflow issues (see more info here and here). "Large" of course depends on the input data as well.

You can see this in action if you run the code to visualize the 12th degree polynomial. The jaggedness and missing points it should hit exactly show a similar concept (which Dash goes into here).

wcraine

Thanks, dylarm !

I think of R as basically magic. Free and all-powerful. I thought I was doing something wrong, since it couldn't possibly be a limitation in R. But, of course, brute-forcing a polynomial in the high teens is going to produce wickedly large numbers to deal with.

I appreciate your time.