*UPDATE: Dan in the comments correctly points out that the numbers I listed are actually double what they should be. All the attempt numbers should be cut in half. Thanks, Dan.*
On September 7, 2008, rookie Matt Ryan made his debut, launching a 62-yard touchdown strike to Michael Jenkins on his first ever NFL pass. It was quite the start for the 3rd overall pick, but while the future was bright for the young signal-caller, nobody expected him to average 62 yards per attempt. He would most certainly come back to earth.
So we can all agree that one pass attempt is not enough data to draw conclusions about a player’s true ability. In his debut, Ryan went on to complete 9 of his 13 passes (69.2%) without an interception; in his second start, he completed just 39.4% of his passes with no TDs and 2 INTs. Again, most of us will accept that a game or two is still too small of a sample. So, what is the point at which we can start to accept the results as indicative of a player’s true talent?
To answer that question, I will use a technique championed by Tom Tango in baseball to determine how long it takes for specific skills to stabilize. The idea is to find out how many attempts (or whatever denominator you are using) it takes before the observed data is half real (skill or true variance) and half noise (luck or random variance).
The Process
To find the point at which the data is half real and half noise, I need to determine where the correlation (r) between two random sets of attempts is 0.5. Let’s take pass yards per attempt as an example. I’ll start with, say, 200 attempts. For all QBs in my set (2000-2009) who have at least 200 attempts, I randomly select two sets of 100 attempts each, calculate the YPA for each set for each QB, and then find the correlation between the two sets. To make sure the correlation coefficient converges, I repeat this process 25 times. I then choose a new number of attempts and repeat the process again, producing a new “r” for each number. For each bucket, we can predict the point at which r = 0.5 by plugging into the formula ((1-r)/r) * Attempts. Below is the table for Yards Per Attempt:
| Attempts | r | n | r = 0.5 |
| 200 | 0.24 | 121 | 634 |
| 400 | 0.37 | 95 | 677 |
| 600 | 0.41 | 78 | 860 |
| 700 | 0.48 | 71 | 763 |
| 800 | 0.49 | 64 | 831 |
| 900 | 0.52 | 61 | 824 |
| 1000 | 0.58 | 56 | 710 |
| 1200 | 0.64 | 45 | 689 |
| 1400 | 0.68 | 42 | 657 |
| 2000 | 0.72 | 30 | 797 |
As you can see, the projected number of attempts for where r = 0.5 hovers around 800. So when a QB reaches approximately 800 pass attempts, our best prediction of his true YPA talent would be half his current YPA and half league average (we could use the average for some other population besides the league if we choose, but league average is generally a good starting point).
The Results
Let’s now look at the results for six important QB metrics.
| Stat | Formula | Stabilizes | Seasons |
| Sack% | Sack / Dropback | around 400 dropbacks | 0.75 |
| Comp% | Comp / Att | around 500 attempts | 1.00 |
| YPA | Yards / Att | around 800 attempts | 1.60 |
| YPC | Yards / Comp | around 650 completions | 2.15 |
| TD% | Pass TD / Att | around 2250 attempts | 4.50 |
| INT% | INT / Att | around 5000 attempts | 10.00 |
To interpret this table, we can say: “At around 500 attempts, a QB’s completion percentage is half real and half noise.” I added a “seasons” column to give a sense of about how many seasons each stat takes to stabilize. For example, Yards stabilizes faster with respect to completions than when compared to attempts. However, it takes a little over 2 years for a QB to pile up 650 completions, compared to around a season and a half to get to 800 attempts, meaning the QBs YPA will actually stabilize faster in real time than YPC.
A bit of a surprise to some is that Sack % actually stabilizes the fastest. While this doesn’t necessarily mean that QBs themselves most control how often they are sacked (offensive line, scheme, opponents, etc. are also involved), it does suggest it. On the other end of the spectrum is INT %, which takes approximately 5000 attempts to stabilize. That means that until a QB plays nearly 10 seasons in the league, his true interception rate is probably closer to league average than his current interception rate. The implication that QBs have much more control over their sack rate than their interception rate is worth further research. At the very least, sack percentage is much more stable than interception percentage, whatever the cause.
Conclusion
This provides an interesting look at what things a QB most controls. Sack rate, yards per attempt or completion, and completion percentage are all things that quarterbacks have sizable control over. Touchdowns and especially interceptions are much more susceptible to luck and other extraneous factors.
Tom Brady, for example, is coming off of a fantastic MVP performance, where his 0.8% INT% lowered his career rate to 2.2%. However, at 4700 career attempts and using 3.0% as the league average, we’d expect Brady to be around 2.6% going forward, a far cry from his incredible season in 2010. On the other side, Eli Manning saw his INT% climb to 4.6%, the highest of his career (3.4% career rate on 3300 attempts). This analysis suggests he’ll be expected to return near league average next season. To make actual predictions, we’d want to do a much more rigorous analysis including many more factors, but this gives us a sense that guys like Brady and Eli are much more likely to see their interception rates regress heavily towards league average than stay at the extreme levels we saw last season.
We can use this type of analysis to look at team-level statistics, or players at other positions. The idea is to help us know when statistics stabilize and become reliable, and when we should take them with a grain of salt (or league average).
This is super gay. Brady will repeat his incredible int performance next year. Can we get a column analyzing Peytons statistical decline the last few years?
This is completely in line with all other research I’ve seen. Fans blame the line for sacks and the QB for picks, but in actuality, the QB is much more responsible for sack rate than INT rate.
There’s been good work done over at profootballreference that shows the same results.
It’s counter-intuitive, but true.
Very interesting research. Are you considering making this a full paper and submitting it to some of the premier sports stats journals?
This is really interesting. Great presentation, too – easy to read. Gives me some things to think about. Thanks for sharing!
Ben, I am not planning on turning anything into a full paper and publishing it. Just trying to put my thoughts out there and hopefully learn something.
Kenos, appreciate it!
Correlation & Causation are two different things. Just wondering if its easy to generate:
– How’s teams’ and QBs’ sack rate look when a good QB switches teams?
– How’s the teams’ and QBs’ sack rate look when a good lineman switches teams?
– How do these metrics look when we take out every QB that has only played as a starter for one team?
(p.s. i know “good” is subjective…sorry)
Its counter intuitive that a QB cannot improve their sack rate with experience, and its counter intuitive that its the fault of the QB not improving with experience.
I think (well now its ‘hey maybe’..) sack rate is a measure of Oline play, Oline play is more consistent, and Oline play overrides improvement in QB play, when it comes to sack rate, and hey maybe the data i requested will show it.
Great work though…
Jimmy, agreed correlation does not necessarily mean causation. Here are a couple studies at PFR that might answer a couple of your questions:
http://www.pro-football-reference.com/blog/?p=4152
http://www.pro-football-reference.com/blog/?p=4395
As far as QBs improving with experience, I did not address that here. Of course, when stats take up to 10 years to stabilize you certainly enter into the zone where players are improving/declining and their true talent level is not static.
And as for O-line play and it’s effect on sack rate and other passing metrics, I do have a couple things I’m looking at. Look for a future post on the topic. Thanks for the comment.
I think you might be understating the stability.
If I’m understanding your calculation correctly, the 800 attempts (r=.49) row of your YPA table means that the correlation between YPA in 400 attempts and YPA in 400 other attempts is .49. But you’re talking about it as if it’s showing that the correlation between YPA in 800 attempts and a QB’s true underlying YPA ability is .49. In fact, it shows that the correlation between YPA in 400 attempts and a QB’s true YPA ability is .70 (sqrt(.49)).
If the correlation between true ability and one sample is r, and the correlation between true ability and another sample is also r, then the observed correlation between the two samples will be r^2, so you want to take the square root of the observed correlation (.49 in this case) to estimate the strength of relationship between one sample and the true ability.
In this case, if you knew a QB’s true YPA ability for certain, then you could predict his YPA over the next 400 attempts with r=.7. If you observed his YPA over 400 attempts and used that to predict his true ability, and his exact true ability was then magically revealed to you to check your accuracy, you could predict it with r=.7. What you’re doing with the data is first you’re using observed YPA in 400 attempts to estimate true ability (r=.7), and then using estimated true ability to predict YPA over the next 400 attempts (also r=.7), so the observed correlation is .7x.7=.49.
If you want your table to reflect the correlation between the observed performance and true ability, you should halve the attempts numbers and take the square root of the r values. Using those numbers, a QB only needs about 160 attempts to have a correlation of r=.5 with his true YPA talent, about one fifth of what you calculated.
On the other hand, there are also ways in which a QB’s performance is less predictive of true ability than these numbers suggest. Observations of a single season (which is what we get in real life) aren’t as informative as randomly selected attempts from throughout a player’s career (which is what you used here), because other variables that influence a QB’s performance (offensive line, receivers, offensive system, etc.) tend to be relatively fixed within a single season but to vary over a career. So in a single season they’ll bias your estimate of a QB’s ability (e.g., Cassel in New England) but with attempts selected from throughout a career they’ll just be noise.
Dan, thank you. You are correct, I will post an update noting this at the top of the page. The relative values are still valid, but all the absolute numbers do need to be cut in half.
Hello
YOU NEED QUALITY VISITORS FOR YOUR: outsidethehashes.com ?
WE PROVIDE HIGH-QUALITY VISITORS WITH:
– 100% safe for your site
– real visitors with unique IPs. No bots, proxies, or datacenters
– visitors from Search Engine (by keyword)
– visitors from Social Media Sites (referrals)
– visitors from any country you want (USA/UK/CA/EU…)
– very low bounce rate
– very long visit duration
– multiple pages visited
– tractable in google analytics
– custom URL tracking provided
– boost ranking in SERP, SEO, profit from CPM
CLAIM YOUR 24 HOURS FREE TEST HERE=> ventfara@mail.com
Thanks, Ferne Horton