The Failure of Football Statistics: Insight from a Pro NFL Analyst
Josh Johnson of the San Francisco 49ers passes during the game against the Houston Texans at Reliant Stadium.
Credit: Michael Zagaris / San Francisco 49ers / Getty Images
In 2007, Neil Hornsby, a British business consultant, launched Pro Football Focus[1]. Enamored of NFL football since discovering Dan Marino in Touchdown magazine in 1984, Hornsby longed for a better way to evaluate players than the opaque box scores the league churns out weekly. PFF was the culmination of his efforts to find better numbers, a vast database of statistics and grades tracking every player for every play, every game of his career. Since the Giants first discovered the strategic advantages of PFF's work in 2009, 11 teams have contracted with Hornsby for custom data and a new perspective on the game. Here, Hornsby talks about the failures of traditional football statistics and what the numbers reveal about who really should have won Offensive Rookie of the Year in 2013, why the Jets should be bullish on Geno Smith, and which player is the most under-celebrated in the NFL.
It's hard not to assume that PFF is just trying to invent a sabermetrics for football. Is that what you're doing?
There's been a lot of people who've wanted to be the Bill James of football — that's what everybody seems to aspire to do, to bring Moneyball-type ideas to football. I just don't think that's possible, because the statistical sample sizes are too small. What we do is performance-based scouting. In the NFL they will send out these guys who have been steeped in football for many, many years, almost since birth, to go and look at college players and tell them whether they're going to make it in the league or if they're NFL players, whether they'll make it on their teams. What they won't do, necessarily, is look at every single player on every single play. They will not tend to do that. In fact, they don't do that.
What they do is say things like, "This guy's a knee-bender. He's sideline-to-sideline." But they don't try and catch that in terms that are necessarily quantifiable. What we try and do is give that information in quantifiable terms. So we try and say that, over the course of a season, having watched all 1,000 players and offensive linemen and all 450 blocking players, that this guy typically makes a lot fewer mistakes than that guy. So what we do is we look at every single player and we try to grade that player on what we believe it to be.
RELATED: The Best (And Least Known) NFL Ranking System
[2]
What are you able to see that professional NFL scouts can't?
Sometimes people get caught up the art of the possible. They say, "I've seen this player make a circus catch once, therefore I believe I can get him to do it 10 times out of 10." But my view would be, "Okay, so you've seen that guy do it once. We want to know how possible it actually is." And typically, that player will do it on three percent of plays over the previous three years. What is it that's suddenly, magically, going to make him do it on 25 percent of plays the next year?
What's wrong with traditional statistics?
A sack seems pretty unequivocal: A defensive player tackled the quarterback behind the line of scrimmage, or he didn't. Well a sack isn't a sack. Not all sacks are created equal. Say a defensive end gets 15 sacks in a season. The next thing I'll say is, "Okay, well, what about hits and hurries?" What's better: An unblocked sack, or a play where you beat two guys, hit the quarterback and the ball flies up in the air and it's intercepted? Football's all about context. You've got to look at the context of each individual player to make sense of it.
Don't your grades flatten that context, though? Should people be suspicious of the claim that a player's entire career can be summed up by one number?
Of course they should. And they'd be right to do so. But you have to look into the grades. You need to look at how those grades are broken down by week. Here's an example: If I were a Jets fan, I'd actually feel fairly positive about [New York Jets quarterback] Geno Smith. If I were a Buffalo Bills fan, I wouldn't feel very comfortable having E.J. Manuel as my quarterback on the basis of last year. What you see with Geno is a significant improvement as the season wore on. With E.J. Manuel what you saw was a fairly static poor performance throughout the season. Even though his worst games were less bad than Geno's worst games, there was no improvement.
You mentioned earlier that NFL teams are seduced by the "art of the possible." Don't you, with all the relentless analysis, suck the magic out of football?
I don't see how it does. I think football is the most magical sport. It is an unbelievably complex, difficult, thought-provoking sport. And I actually think we add to the whole thing by revealing the contributions made by every player. We bring the game back to all the players on the field. For example, I struggle with the concept that the best players in the NFL are always quarterbacks or running backs. Offensive Rookie of the Year last season was [Green Bay Packers running back] Eddie Lacy. And he did a sensational job. But I just don't know how you could look beyond [Detroit Lions offensive lineman] Larry Warford. The way he played as a third-round rookie was absolutely outstanding. I don't think anyone mentioned him. Because he's an offensive lineman, and why? Because there aren't any stats.
I think the NFL is getting better about these things. I think most people realize that there are issues with ways we evaluate players, because real injustice can be done. The classic example from last year was: How is [Tampa Bay Buccaneers linebacker] Lavonte David not a Pro Bowler? Not just his stats, but the way he played — that is just incredible. Most things in football are just shades of gray. But Lavonte David not being a Pro-Bowler — that's just not one of them. That's just utter, unmitigated stupidity.
References
- ^ Pro Football Focus (www.profootballfocus.com)
- ^ RELATED: The Best (And Least Known) NFL Ranking System (www.mensjournal.com)