Two questions regarding student-created models (3.6)

bkirk272

The last question of Notebook 3 encourages students to develop their own model. It's not explicitly stated that this model should be using default_rate as the response variable, but is that the intent of the question? I've had some students create interesting models using grad rate as the response, but I've been advising them that is going a little off theme from the rest of the analysis.

Also, one of my students produced a model using hbcu as a predictor, and R handled that no problem, producing a regression output like normal. We were a little perplexed at how R was using a categorical variable in this manner. I know that I've seen other instances where a categorical variable is "converted" into a quantitative one by using 0 = "No", 1 = "Yes" for example; is this what R is doing here?

dashiell-young-saver

Using default_rate as the response variable is the intent. This will also align with the prediction competition in Notebook 4, where they'll be evaluated on how accurate they depict default rates. However, making models with other response variables is certainly a worthy and interesting exercise! So, I think, it's really up to you 🙂
Yes, binary variables like hbcu are implicitly converted into 0 & 1 values. So, the coefficient value for hbcu represents the association with default rates for schools with a "1" for hbcu (which is all the HBCU's), controlling for the other variables.

bkirk272

I haven't tried this yet, but would the model work with other categorical variables?

dylarm

bkirk272 In a sense, yes. R will convert categorical variables into what it calls "factors" and then each level of the factor will be given a coefficient in the model. One use of it that I see the most often is to do something like default_rate ~ pct_PELL:region so that each region receives its own coefficient for pct_PELL.

bkirk272

dylarm Okay this makes sense as a way of showing the "slope" for pct_PELL within each region (while other variables are controlled). What if region is used as a predictor variable on its own? Then each region is given a coefficient, but it's unclear to me what those coefficients actually represent. For example: grad_region_model <- lm(default_rate ~ grad_rate + region, data = dat) creates the following output:

Coefficients: (Intercept) grad_rate 12.8580 -0.1495 regionMidwest regionNortheast 0.4455 1.0749 regionRockies & Southwest regionSouth 0.8385 2.1006 regionTerritories 1.7695

What do those region coefficients actually mean? Are they slopes?

dylarm

bkirk272 The forum seems to bork longer code blocks 😆 But I get what you mean. In that case, you can think of them as intercepts. So there's the raw "intercept" that's kind of the reference point, but then each categorical variable receives a coefficient that effectively gets added to that for each level.

bkirk272

dylarm So how are those intercepts interpreted? Certainly no longer "As [x] increases by 1, default_rate increases by [coefficient]." If they can't be interpreted in context, how should they be interpreted as indicating strength of associations?

dylarm

bkirk272 This is actually a little bit of a more advanced topic when it comes to linear models and their representation in R in general. It's a grad school level rabbit hole.

If you feel up to it, I recommend reading this Cross Validated post along with its answers for a more thorough explanation, or this Stack Overflow post for a more simplified version, or this simpler Cross Validated post that I just found. Or you can see the complexity I mean from this R-bloggers post. Briefly, interpreting in context means that you have to know which one R is using as its reference, because the values for the other ones are based off of that, but sometimes only if it's converted to an actual factor (categorical variable) first, rather than left as a character string in the data frame. And in certain contexts, they can also be viewed as simply the mean of that particular group. It's… complicated, to say the least. I have to brush up on it every time as well.

As far as interpreting strength, if I understand you correctly, that's a bit easier. Using the table from the third link up above, here's the output of using the summary() command on the model:

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 5.565556   0.145851  38.159  < 2e-16 ***
animalfox   0.806634   0.131198   6.148  8.6e-10 ***
animalpig   0.010635   0.131259   0.081   0.9354    
animalwolf  0.806650   0.131198   6.148  8.6e-10 ***
weight      0.007946   0.003815   2.083   0.0373 *  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

The Pr(>|t|) is the p-value of that particular coefficient, and the asterisks/dots/blanks on the side are a quick quick to tell you what particular value of alpha it might fall under.

If you wanted something like an R² value for each coefficient, then, at least with my current understanding, you may be out of luck. The p-value for each coefficient is typically what's used, and the entire model is judged based on its p-value and Multiple-R² (or Adjusted R², log-likelihood, Akaike information criterion, etc.).

Hope I haven't confused you too much! 😄

bkirk272

dylarm Okay, thank you. This is helpful, if a bit perplexing. I wonder about the legitimacy of using categorical variables in the prediction challenge, then, if it's a little nebulous what exactly R² comes to mean when categorical variables are used as explanatory variables.

dylarm

bkirk272 R² is still the coefficient of determination that explains the proportion of the variation in the response variable that is explained by the independent/predictor/explanatory variables, regardless of whether they're numerical or categorical. Just the usual sentence frame needs to be modified from

<R²>% of the variability in <response variable> can be explained by its linear association with <explanatory variable>

to something like

<R²>% of the variability in <response variable> can be explained by a linear combination of <explanatory variables>.

So I suppose to use your original example, it might be something like: (I didn't actually check what the R² value would be)

30% of the variability in loan default rate can be explained by a linear combination of graduation rate and HBCU status.

tspence

I had a question about this too because I have a student who used "city" and "state" as two of his predictor variables. I have several problems with this because I don't really know how to interpret the output (and I'm sure he doesn't either) and the output is pages and pages long because it separates every town and state. On the flip side, he got a really high R² value, so maybe he's on to something?

I don't know what kind of feedback to give him….

dylarm

tspence So state might be okay, because there are multiple universities in each state and ideally each state would be present in both the test and train dataset. I've not checked this, but it's also not something that's guaranteed in general.

But city should probably be avoided, because it's essentially overfitting the model. I presume most cities only have one university, so the ones that have two or three are likely to be highly correlated with each other (think: San Francisco vs rural Wyoming). And we want to avoid overfitting. 🙂

I know I found the average number of universities per city at one point, I'll have to dig that number up again…

dashiell-young-saver

tspence To expand on the great answer from @dylarm , I imagine that your student is in Notebook 3, correct? If so, using city will help them get a really good R-squared value. But, as they'll discover in Notebook 4, this R-squared value is artificially high. This is part of the learning process as they get into Notebook 4.

To see why, let's take an even more extreme example: imagine the student includes the university name as a predictor in their model. R will build a coefficient value for every University name. Since every university name in the dataset corresponds to a single university and its unique default rate, all of these coefficient values will almost perfectly predict their university's default rate. Because students are using the same dataset to fit their model and evaluate their model (get R-squared) in Notebook 3, their model will make almost perfect predictions. The R-square will be extremely high. This will seem like a good model!

However, in most cases, we're not interested in building models that predict the data we currently have. We want to predict future data or unseen data. In Notebook 4, we do a common data science practice: we randomly split our full dataset into a "training" dataset and a "testing" dataset. We fit our model on the training dataset, and then we evaluate its R-squared on the unseen testing dataset. This way, we simulate our model's ability to learn from trends and make good predictions on new (unseen) data. A model that uses university name or city name will not generalize well to new data, as many university or city names that appeared in the seen (training) data will not appear in the unseen (testing) data (or vice-versa). When finding the R-Squared value on the testing data, the program will throw an error. This is a sign that your model is extremely "overfit" - so complex that it's fit incredibly well to the seen (training) data, but doesn't learn the broad patterns that generalize well to unseen (testing) data.

Apologies for the long-winded explanation, but I hope this helps! Let me know if I can clarify.

tspence

dylarm Right. They don't get to the overfitting concept until Notebook 4. I forgot about that. If you find that average, I would be curious, but don't hunt it down just for me.

tspence

dashiell-young-saver Yes, that is helpful. Thank you!

dylarm

tspence I was definitely thinking about Notebook 4 instead of 3, oops 😅

Anyway, there are approximately 1.44 universities per city, and ~20 per state.