Earlier today, ESPN released (some) details of their brand new rating system for quarterbacks dubbed Total Quarterback Rating, or Total QBR (or even further abbreviated, just QBR), aimed at replacing the popular yet flawed Passer Rating. So what is it? And is it a worthy replacement?
To answer that question, let’s first take a look at what is currently out there, which will let us compare Total QBR and pinpoint the major differences.
Current QB Evaluation Systems
Passer Rating: Introduced in 1971, combines 4 basic passing statistics: completion percentage, yards per attempt, touchdown percentage, and interception percentage. The rating involves arbitrary floors and ceilings on each component and then combines them into one final number. Passer Rating does not include sacks or rushing plays for QBs.
DVOA and DYAR: Football Outsiders proprietary statistics. DVOA is presented as percentage above (or below) average. For example, a 26.0% rating means that player performed, on average, 26% better than an average player would have give the same opportunities. DVOA is based on ” down and distance, field location, time remaining in game, and current scoring lead or deficit”, as well as being adjusted for defense (that’s what the “D” is for). While DVOA is a rate stat (i.e. calculated on a PER PLAY basis), DYAR is a cumulative, or counting, stat that reinterprets DVOA by putting it on a scale of yards above replacement player. Therefore, all plays are added together instead of averaged like in DVOA.
EPA and WPA: Brian Burke, at AdvancedNFLStats.com, uses two approaches that somewhat resemble DVOA. His explanations are likely better than mine (click the links), but I’ll summarize here. EPA, short for Expected Points Added, is based on Expected Points (EP). EP is based on down, distance, and yardline, and is essentially the (net) expected points an average team can expect in that situation. This includes future drives (so EP can be negative, which it is in certain situations deep inside your own territory) and is looking for the “next score”. EPA is simply derived by subtracting the EP at the end of a play from the EP at the start of a play. WPA, or Win Probability Added, is exactly the same as EPA, except it is based on Win Probability (WP) instead of EP. WP is based on down, distance, and yardline–just like EP–but also includes time remaining and score. EPA and WPA are both cumulative stats (like DYAR), but can be divided by the number of plays to produce a rate stat (akin to DVOA).
All of these statistics attempt to define what a quarterback (or in many cases, any player) accomplished. Each play that the QB was involved in gives the QB full credit (or blame), there is no division of credit. For example, if a QB throws a 3-yard pass that the WR runs for another 20 yards, that is the same as the QB throwing a 23-yard pass where the WR is immediately tackled. This will be important later.
I know, you’re thinking “wow, that was a lot of information, couldn’t he have just compiled it into a nice little chart?!” And the answer is yes, so for those of you who prefer charts (like me), enjoy (if you click the image twice, it should come up bigger):
Total Quarterback Rating
Here is a not-so-brief summary of Total QBR as I understand it. I urge everyone to read the ESPN article by Dean Oliver (link in the first paragraph).
Total QBR starts with a baseline of Expected Points. In this respect, it is nearly identical to EPA as AdvancedNFLStats calculates it, though it looks like ESPN is adding in a few more factors. As far as I can tell, those are field surface (grass or turf, I presume), time remaining, timeouts remaining, and home team. Time remaining, I believe, is mostly to account for the half ending (and thus terminating the possession).
The next step in Total QBR is to divide up credit between the QB and the other 10 players on offense. This is a pretty big deviation from the other major QB rating metrics out there. To use my previous example, a 3-yard pass that becomes a 23-yard gain due to the WR running for 20 yards after the catch is no longer the same as a 23-yard pass completion where the WR is tackled immediately. ESPN, through their detailed game tracking, is trying to determine how responsible the QB is for the result of the play based on things such as the accuracy of the throw, the pass protection, the defense, etc. The article does not go into much further detail about how they determine what portion of credit or blame to partition to each player.
So the QB has started with a number of Expected Points Added based on the result of the play, let’s call this EPA. He then is given some portion of this credit (or blame), let’s call this C. So his current value for this play is EPA * C. If the EPA is 1.2 and he gets half the credit, he currently stands at 1.2 * 0.5 = 0.6 EPA.
Now, ESPN adds in what they call a Clutch Index. As far as I can tell, this is exactly the same as what baseball calls Leverage Index. All Leverage Index, or Clutch Index, is doing is telling you how important a specific situation is. Technically, it tells us, on average, how much WPA we can expect on the next play. Let’s use a quick and extreme example. Let’s say there is 1 second left, and the Packers have the ball on the 4-yard line, down by 4. For argument’s sake, the Win Probability in this situation is exactly 50% (I don’t know what it really is, but it doesn’t matter). On the next play, the Packers are either going to score and win the game, moving their WP to 100% for a WPA of 0.50, or they are going to fail and lose the game, resulting in a WP of 0% for a WPA of -0.50. On average, the absolute value of the WPAs is 0.50, an extremely large number. And let’s say an average play is worth, on average, 0.05 WPA (again, numbers only for demonstration purposes). And finally, a play at the end of a blowout is worth only 0.01 WPA on average.
Leverage Index puts this average change in WPA based on the situation on a scale comparing it to an average play. So our average play sees a change of 0.05 WPA, and it’s Leverage Index is 1.0–exactly average. Our high pressure play saw a change of 0.50 WPA on average, or 10x as much as an average play, so it’s LI is 10.0. And our play at the end of a blowout caused only 0.01 WPA change, or 20% as much as average, so it’s LI is 0.2, very low. According to the ESPN article, most Clutch Indexes will fall between 0.3 and 3.0.
Okay, so back to our example. Our fictional QB currently sits at 0.6 EPA. But let’s say this was an extremely important play, so the Clutch Index (CI) was 3.0. Total QBR multiplies the EPA by 3.0, so the play is now worth 0.6 * 3 = 1.8 EPA for our QB. Our formula now sits at EPA * C * CI.
The final step is to divide by the total CI for all plays, so first we sum up each play’s EPA and factor in the credit, C, and the Clutch Index, CI. So we have SUM(EPA * C * CI) for each play a QB is involved in. And then we divide by the total CI, SUM(CI). So a QB involved in a lot of important plays will have a SUM(CI) above 1.0 and vice versa. So our final formula: SUM(EPA * C * CI)/SUM(CI). Oh yeah, and then they do some simple algebra to put it on a 0-100 scale with 50 being average.
So What’s New?
Alright, so let’s look at the new things that Total QBR is actually bringing to the table.
1) Adding in field surface, time remaining, home team, and timeouts to the Expected Points calculation. This is a small change.
2) Attempting to divide up credit between QB and teammates, largely based on new, proprietary data that ESPN tracks. This is a bigger change.
3) A Clutch Index factor. What is this actually doing? Not much really. By multiplying EPA by CI, you are essentially giving more credit to plays that impact the outcome of the game more…or exactly what WPA does. So EPA * CI = WPA (not exactly, but essentially). That leaves our formula as: SUM(WPA*C)/SUM(CI). The SUM(CI) term is accounting for players who are given more opportunity to have a bigger impact. So making big plays in big moments helps, but you don’t get credit simply for having the chance (more on this in a later post, as it is an important yet confusing point).
Summarize This PLEASE
Yes, I know, a lot of technical jargon to get the point. But here it is: Total QBR is simply a Win Probability Added statistic with an attempt to divvy up credit on each play (as opposed to just giving the QB full credit), and with situational opportunity removed.
Is this any better than what’s out there? It is better than Passer Rating, yes. It is a rate stat, so we are comparing it to DVOA, EPA/Play, and WPA/Play. It is most like WPA/Play or DVOA, with the main difference being that they attempt to give a QB the correct amount of credit. Removing situational opportunity is simply a matter of preference, but the way they arrive at it is somewhat awkward. This could be made much simpler by just starting with WPA instead of EPA and leaving out the Clutch Index factor (but keeping it in the denominator).
In the end, the usefulness of this stat depends on what you want to measure. This is long-winded enough, so I’ll save my personal preference for another post. For now, soak it in, and be grateful that we’re at least on the right track towards a better mainstream representation of quarterback play.