BCS Series: Review of Anderson & Hester ratings
In the third installment of my review of the BCS computer rankings, I will take a look at the ratings of Anderson and Hester. For starters, they have a great tagline on their website: “showing which teams have accomplished the most”. For those of you that have been following, you know my stance on how teams should be judged for inclusion to the BCS title game and this fits perfectly.
Anderson and Hester don’t give many details about their system, but they do highlight four ways in which they believe their ratings to be distinct. Let’s take them one by one.
1. Unlike the polls, these rankings do not reward teams for running up scores. Teams are rewarded for beating quality opponents, which is the object of the game. Posting large margins of victory, which is not the object of the game, is not considered.
This follows from their tagline as the way to measure what teams have accomplished is through wins and losses, “which is the object of the game.” This is exactly right.
2. Unlike the polls, these rankings do not prejudge teams. These rankings first appear after the season’s fifth week, and each team’s ranking reflects its actual accomplishments — on the field, to date — not its perceived potential.
If you caught my review of the Billingsley ratings I harped on this point quite a bit. If you are trying to reward teams for their season, you cannot use anything but their performance in that season.
3. These rankings compute the most accurate strength of schedule ratings. Each team’s opponents and opponents’ opponents are judged not only by their won-lost records but also, uniquely, by their conferences’ strength (see #4).
This is where we run into a problem. Why do you need to use the conference strength to rate teams? According to this site, “A&H also weighs strength of schedule according to conference strength, which is also measured by the conference’s opponents and opponents’ opponents.” Teams should be judged by the teams they actually play. Their conference should have no bearing on their rating outside of the fact that they play many teams within their conference. However, the rating system should simply look at the games each team played, and adjust accordingly. For instance, what if an SEC team plays Alabama but avoids Vanderbilt, while another plays Vandy but avoids the Crimson Tide? Do they both also get the same conference adjustment? That is clearly not right. The team that played Alabama clearly faced a tougher schedule. Simply look at each individual team and use only the opponents on their schedule to determine the strength of schedule they faced.
The previous web site I linked to also mentions that “as far as strength of schedule, A&H takes into account both a given team’s opponents and the team’s opponents’ opponents.” If this is true, this is an acceptable way to look at strength of schedule. Looking at wins and losses is definitely biased, as you will find a large spread in talent between teams with the same record based on the schedules they played. Ideally, you would use something like an iterative process to look at opponents’ opponents’ opponents (the third level deep), the fourth level, and so on. You do this until you reach convergence for the strength of schedule (meaning that going another level deeper only changes your SOS rating less than some previously determined small amount). However, in practice, going just two levels deep is likely sufficient (though this is worthy of testing).
4. These rankings provide the most accurate conference ratings. Each conference is rated according to its non-conference won-lost record and the difficulty of its non-conference schedule.
This is a claim the authors make, and it may or may not be true. In any case, the BCS rankings are concerned with teams and not conferences so I will not comment on it here.
The main framework of the Anderson & Hester rating system is a good one. They incorporate wins and losses with a strength of schedule based on two levels of opponents’ records, while focusing solely on the current season. The one odd part of the system is the inclusion of conference strength in the ratings, which is unnecessary once a team’s schedule has already been considered. Overall, this seems like an adequate but flawed system. Learning about exactly how the conference strength is factored in is key to determining the quality of the system.