Last year, I introduced the Achievement S-Curve. The idea behind it was that teams should be rewarded for their season based on their wins and losses and the strength of their schedule. This is in opposition to the other camp of evaluating and seeding teams for the tournament, where teams are judged based on who is the “best” regardless of record. I discuss this dichotomy in further detail in this post from last year.
Methodology
The result was my Achievement S-Curve, and I’m bringing it back for a second go-round this year. I explained the methodology last year, but I’ll give a quick summary here:
- I believe a system should satisfy two main constraints: (1) no win should hurt a team and no loss should help a team and (2) the same record against the same schedule should be rated equally, regardless of which teams the wins and losses came against. (In my original post, I had a third “constraint” that the number of games shouldn’t inherently help or hurt a team. More on this below.)
- To determine a team’s Achievement Score, I first determine the difficulty of each game. I find the probability of the opponent beating a baseline team at that location (home/away/neutral), based on the true strength of the opponent (in other words, I evaluate opponent strength based on who is “best” not who is “most deserving”). I use my own rating system to make this determination, but you could substitute any rating system you’d like (e.g. Sagarin predictor rating).
- The next step is to adjust each team’s score based on whether they won or lost. If they won the game, I add the difficulty of the game to their score. For example, North Carolina at home against a baseline team might win 95% of the time. Winning at North Carolina, then, would add .95 points to a team’s Achievement Score. Losing, conversely, would subtract the opposite of the difficulty from their score. Losing at North Carolina, to continue the example, would cost a team just .05 points.
- For my “baseline team”, I have made a change. Last year, I used an average D-1 team as the baseline. This time, I am changing the system to use an “average tournament team”, which is roughly equivalent to a team that would beat an average team about 85% of the time. What to use for a baseline team is mostly a matter of preference. The reason for the change is that using an average team ended up giving too much credit, in my opinion, for beating slightly above average teams (in the 75-150 range) and not enough credit for beating the top teams. Since we are using this system to determine tournament teams, I thought the move to an average tournament team would be more useful.
One issue I have is trying to decide whether to use total score or average score. If I use total score, I run into the problem that good teams are slightly helped by playing more games. As long as a team is better than the “baseline” team, they should be expected to increase their Achievement Score on average with every game played. The opposite is true for teams that are worse than the baseline team (their score will be helped by playing fewer games). Going with average score (Achievement Score per game) solves that problem, but creates another one by violating one of my constraints listed above in that a team can lower their score with a win or raise their score with a loss. Consider a team that has an average AS of .70 and then they play and beat a weak team. That win will get them less than .70 points and so their average will drop, say to .65. So by playing and winning a game, they have now dropped below another team that didn’t play. This doesn’t make much sense to me and so I have chosen the lesser of two evils and am using total score. (If anybody has a solution, let me know in the comments below.)
The Achievement S-Curve
This version includes all games through January 15th. I determined the automatic bid for each conference by who my system rates as the “best” team, since I thought this would be the team most likely to win the conference tournament. Teams given automatic bids are listed with their conference next to their name. As always, click (sometimes click twice) to englarge.
The full Achievement S-Curve can be found on Google Docs by clicking here.
Biggest Surprises
Let’s take a look at which teams’ placement probably have you scratching your head.
- Illinois (ASC 2-seed; Bracketology 6-seed): Illinois has played a tough schedule overall, with their 3 losses coming away from home against very good teams (N-UNLV, N-Missouri, @Purdue). Their record is better than their talent, as a lot of their wins are close. However, compare them to North Carolina (ASC 4-seed; Bracketology 2-seed) and I don’t see how you can arrive at the conclusion that Illinois should be 4 seed lines lower. Both are 15-3, with all 3 losses being away from home against top-30 KenPom teams. So let’s call that a wash and look at the wins: UNC has 2 great wins against Michigan St. and Wisconsin, and 5 other wins of note: Texas, @UNC-Asheville, Long Beach St., Miami (FL), and South Carolina. Their remaining wins were essentially gimmes. Illinois has a huge win against Ohio St. and an underrated win @Northwestern. They then have 5 other wins to match UNC’s and then 2 other decent wins that are better than the Tar Heels: St. Bonaventure and Nebraska. But you know what, we don’t even need all of that analysis. In reality it comes down to this: the two teams have the same record (15-3) and Illinois has played a much tougher schedule (11th vs. 46th).
- Murray St. (ASC 3-seed; Bracketology 7-seed): Teams at the extremes are always the hardest to evaluate and an undefeated team against a soft schedule certainly fits that bill. I have Murray St. with the 292nd-ranked schedule to date, but they haven’t dropped a game. Their schedule will only get worse from here on out, but victories @Memphis, @UAB, and N-Southern Miss help bring up their overall total.
- Middle Tennessee (ASC 7-seed; Bracketology 13-seed): Similar to Murray St., just not as extreme. Wins @UCLA and vs. Belmont help but I think it’s the middle tier of their wins that isn’t getting enough credit. Remember, strength of schedule isn’t just your opponent but location too (not just WHO you play but WHERE you play). The Blue Raiders have 5 other respectable road wins after their big win over UCLA, plus a neutral site victory over Mississippi. I think they’re underseeded.
- Florida (ASC 11-seed; Bracketology 4-seed): Florida is the opposite case of Middle Tennessee: most of their wins are at home. Their best win is @South Carolina, and the rest of their wins are at home or neutral sites against mediocre teams (next best win: Florida St.). Looks like Florida’s getting some special treatment thanks to the name on the front of their jersey.
- Vanderbilt (ASC 12-seed; Bracketology 4-seed): I’m guessing Vandy is getting a bump due to the rule that the committee can evaluate teams differently due to injuries/suspension. The Commodores have played 10 of their 17 games without star center Festus Ezeli. Three home losses, though, have really stung, and one of those (Indiana St.) came with Ezeli on the floor.
- Cleveland St. (ASC 12-seed; Bracketology OUT): One of those bad home losses for Vanderbilt was Cleveland St. I have them as one of the last at-large teams in, while Lunardi has them out (and not listed among the first 8 teams left out). They have 4 losses, some terrible (Youngstown St., N-Hofstra), but 5 strong road wins led by the big win @Vanderbilt.
One big takeaway from all of this is that location is a huge factor. The difference between switching from a home game to a road game in difficulty is akin to switching from playing the 140th-ranked team to the 40th-ranked team. Said another way, the difference between playing @ Louisville (my 40th-ranked team) versus playing Louisville on your home court is the same as the difference between playing @ Louisville and playing @ UTEP (my 140th-ranked team).
Comparison to Bracketology
Overall, my completely objective and automated system provides very similar results to Bracketology. ASC agrees within one seed line on 32 of the teams, and within two seed lines on 46 teams. Three of my at-large teams are not listed in Lunardi’s bracket or his first 8 out teams, but all 3 of those are in the First Four play-in games. All of Lunardi’s at-large teams are at least in my first 8 out. Where we differ, I think I’ve showed that it’s either a matter of preference (i.e. “best” versus “most deserving”) or a matter of flawed analysis as in the cases of Illinois and Florida.
The big advantage of the Achievement S-Curve system is that it is objective and transparent (I think the only part of my system that I haven’t shown here is the opponent strength calculation, but I can easily do that or that part of the system can easily be replaced by KenPom or Sagarin ratings). There may be some things that you disagree with and would do differently–maybe you’d like to give more credit for big wins or perhaps give credit for close losses–but you can start with this system and tweak it to match your preferences. The positive of doing it this way is that you are objective and consistent in your application.
Conclusion
Without something like this, we fall into the trap of just highlighting parts of a team’s profile: their good wins or bad losses or top-100 record. Heck, even I couldn’t resist in my analysis above and I’m writing a post about not doing it. It’s nearly impossible for a human to objectively evaluate all of a team’s 30+ games, let alone apply that method consistently across 300+ teams. Instead, let’s create the system and let the results come from that. I know Murray St. and Middle Tennessee agree with me.