The "Extra Crispy" Formula
If you are a regular reader of this blog, you have heard me preach innumerable times that there is nothing mystical about computational models, be they models of college football team performance or models of hurricanes. The only thing that differentiates a "computer" model and a human "gut" feeling is the amount of data used and the amount of bias awareness built in.
So in a previous edition of this blog, we talked about my most recent attempt to acknowledge some potential biases in DUSHEE. These tweaks resulted in the "Extra Crispy" formulation of DUSHEE which was supposed to correct some of the perceived biases of the "Original" formulation: overrating Group of 5 WTF teams, overemphasizing outlier performances when a team does far better or worse than they have on average over the rest of the season.
But if you've noticed, and no doubt you have, I kinda ignore the "Extra Crispy" formulation, at least until bowl season, when it reemerges. I ignore it because it severely punishes teams with weak SoS, which means it severely punishes Group of 5 schools. In other words, I think it has swung the pendulum too far the other way. The "Extra Crispy" formulation would have heavily penalized the MWC-era Frogs for being in the MWC. And that rankles my own personal bias ... a bunch of Power 5 blue bloods in the top 25 is boring. And, as I've argued before, if you secretly replaced Coastal Carolina's roster with the New England Patriots and had them play a Sun Belt Conference schedule, they'd still be the best team in college football, even if they were playing teams like Texas State and South Alabama every week.
The DUSHEE methodology compares Team A's performance against Team B to how the rest of Team B's opponents did against them. And if most teams play a random sampling of teams throughout the distribution of college football teams (i.e., some good, some bad) then the methodology works pretty well. The inherent problem, of course, is that college football scheduling is not a random sampling; teams in good conferences play a disproportionate number of good teams and teams in bad conferences skew bad. And if the best team in a bad conference beats up on their conference opponents but never plays an opponent from a good conference, the basis for comparison between that team and teams in that better conference becomes weak.
Case Study: The Utah State Aggies
Let's look at the 2018 Utah State team. DUSHEE thought very highly of Utah State last season, ranking them 9th overall at season's end (Original Formulation). Their game-by-game Point Differentials and opponent strength are shown below.
Opp: @ MichSt NMSU AFA | @ BYU UNLV @ Wyo | UNM @ Haw SJSU | @ CSU @ Boise N UNT
PD: -5.92 30.00 12.60 | 31.73 21.00 6.90 | 31.50 35.50 24.00 | -9.60 4.67 52.00
YD: -75.92 69.70 59.70 | 69.91 120.90 -144.90 | 271.00 175.83 245.60 | -250.20 17.33 345.70
OF Score: -7.62 23.37 11.29 | 24.53 19.85 -2.41 | 34.10 32.17 27.87 | -18.50 3.95 51.38
SoS: 5.72 -23.47 2.12 | 5.80 -14.57 -0.68 | -17.78 -8.98 -20.57 | -13.47 15.26 4.90
Utah State played the 118th toughest schedule out of 130 in FBS last season. They racked up big point differentials largely on the weakest of those opponents: New Mexico State, UNLV, New Mexico, Hawaii, San Jose State. Half of their schedule was against teams that were more than a touchdown worse than an average FBS team and against those teams they averaged a PD of 22.1 against those teams. One thing that good teams should do is crush bad teams, and Utah State did that, save for squeaking by a bad Colorado State in week 12.
The Aggies also played four slightly above average teams (Michigan State, Air Force, BYU, and UNT) and the results were mixed with Utah State crushing BYU and UNT, a solid 10-point win against Air Force and losing by a TD against Sparty.
Utah State played one team that was a TD or more better than an average team -- Boise. And in that game, Utah State performed like a slightly above average team, losing to the Broncos by 9, which was 4.67 points better than the rest of Boise's opponents did.
There is evidence that Utah State was a very good football team last year. With the exception of CSU, they crushed the bad teams on their schedule and they crushed two of the five mediocre teams they played.
But the flip side of that coin is that Utah State were just okay against the other 3 mediocre teams and the one legitimately good team they played. So is that the resume of a top 10 team in the country? If A&M or LSU (the teams just above and below USU in the DUSHEE rankings, who also happened to have the 1st and 2nd toughest schedules according to DUSHEE) had played the same schedule, would they have had similar results?
And Now the Tweaks
So if it seems like Utah State was rewarded disproportionately for beating up bad opponents, what do we do? What the Extra Crispy Formulation did before was simply add the opponent's strength to the team's score against that team. In hindsight, that was a lazy cudgel, using a flame-thrower to correct a problem that requires a delicate scalpel. Now, we nudge the individual score with the opponent strength divided by the number of games played. I have also added a reward for games played on the road (acknowledging that playing on the road is statistically about 3 points harder on the road than at home).
So if we were to take Utah State's DUSHEE (Original Formula) point totals and show the adjustments now made by the Extra Crispy formula, we get:
Opp: @ MichSt NMSU AFA | @ BYU UNLV @ Wyo | UNM @ Haw SJSU | @ CSU @ Boise N UNT | Adjustment Sum
OF Score: -7.62 23.37 11.29 | 24.53 19.85 -2.41 | 34.10 32.17 27.87 | -18.50 3.95 51.38 |
Outlier Adjustment: +1.34 -0.06 +0.05 | -0.11 +0.06 +0.22 | -0.08 -0.25 +0.07 | +2.28 +0.28 -2.83 | +0.97
SoS Adjustment: +0.48 -1.96 +0.18 | +0.48 -1.21 -0.06 | -1.48 -0.75 -1.71 | -1.12 +1.27 +0.41 | -5.47
Home Field Adjustment: +1.52 -1.52 -1.52 | +1.52 -1.52 +1.52 | -1.52 +1.52 -1.52 | +1.52 +1.52 0.00 | +1.52
Total Adjustment: +3.34 -3.54 -1.29 | +1.89 -2.67 +1.68 | -3.08 +0.33 -3.16 | +2.68 +3.07 -2.42 | -3.17
XC Score: -4.28 19.83 10.00 | 26.43 17.17 -0.73 | 31.01 32.69 24.70 | -15.82 7.02 47.44
Over a large population of games, one would expect the Outlier Adjustment for poor performances to balance the great performances over the course of a season, but since we have a sample size of 12 here, it doesn't. In this case, the Aggies two negative outliers (Michigan State and Colorado State) were slightly more "outliery" than their one positive outlier (North Texas). I am not sure if the Outlier Adjustment is large enough (e.g., should the "aberrational" effect of being more than 15 points better against North Texas than they were anybody else be "corrected" by more than 2.83 points?) But for now I'm keeping it where it is.
Overall, Utah State's ranking would not have changed based on this new set of corrections, not because their score didn't drop (it did) but because it did not change the ordering of teams once all of them were "corrected" too. But perhaps the corrected formula comes a little closer to judging Utah State accurately?
What say you, Frog rabble? Are the corrections too small? Too big? What else should be considered, taking into account that I'm probably not going to be willing to look at any additional stats beyond points and yards because this thing already takes entirely too much time to piss around with. Even if I love pissing around with it so.