Do we overuse data-based analysis?

I'm back on my "thinking in writing" BS

It’s often been said that the NFL is a year-round sport. I’m of the mind that was probably exaggerated at some points in the past decade or two, but that it’s becoming truer and truer with each passing year.

I’ve been working on some other projects over the past month or so, and I have to say, I’ve never felt this far behind by early March. I have a list of things I want to write, and it’s all been fluttering around my mind over the past few weeks. I have prospect research I want to do. I have reactions I want to offer — to stuff like D.J. Moore moving to the Bears — and the knowledge that free agency begins this week and there’s going to be so much more to react to. Best ball is in full swing. ADP is being tracked, values and fades identified. The Super Bowl was one month ago, today. It’s crazy.

I’m at the point where it feels crazy that I set aside time after the season to work on other projects. Turns out, I should have kept that time set aside to do fantasy analysis. At any rate, I’m going to do something today that I don’t usually do, which is just write. I mean, that’s not totally true, it’s sort of my style to just write. But let’s put it this way: usually I title the post before I start — something not everyone does, but for me it’s a mechanism to try to keep my thoughts moving in one direction — but today I’m doing no such thing. I’ll title this one when I get done.

I want to start here, which is a thought I’ve chewed on for a few weeks:

Ben Gretch @YardsPerGretch

I’ve been a data-based analyst as long as I’ve done this. The industry has shifted recently to where some people are far too certain something has to be in past data to project it. The biggest known in all NFL analysis is change is inevitable. Chaos is the rule.

Kate Magdziuk @katemagdziuk

Justin Fields can have disappointed to date in his career 𝒂𝒏𝒅 still be a star in the making. Both can be true. 🫢

For anyone who has been around here for a little while, those final two sentences come as no surprise. We acknowledge the chaos in the NFL here, and we try to find strategic ways to operate within that framework, adjusting for market valuations at a given time. The bottom line is how best to play that changes, because the market changes. What is valued — including both overvalued and undervalued — shifts over time. Things that were previously undervalued get more expensive, and then inevitably some will make the mistake of thinking a new, proper valuation equals overvalued simply because they are anchoring to a past price that was, again, an undervaluation. The inverse happens, as well. It’s all the same as any other market; there’s a whole fancy term called market dynamics that more or less encompasses all of what I’m referring to.

The first part of my tweet above relates to something I think is pretty important to the current market dynamics in fantasy football. I’ve talked in Stealing Signals over the past couple years how the weekly data I cover has become more ubiquitous, and the types of data the industry uses to formulate its offseason opinions is shifting that way, too. I’ll be the first to say that at times I feel like the game is passing me by, at least to a degree; that there are tweets I see and research I hear referenced that looks really useful, and it’s newer-age stuff I haven’t even begun to dig into. I’m 35 now, with kids that are a little more grown than most 35-year-olds, and I get it a lot in my personal life as well when I find out some new, more efficient way to use my TV from my child. Is this what it’s like to be a dinosaur? If the game hasn’t passed me by yet, that shift has at least started.

And yet, one of the interesting perspectives I think I can bring is, gulp, experience. When I refer to newer-age stuff, one of the things I’d argue is I see a lot of stuff that presents itself as newer age, but is in actuality a rehashing of something that was debunked by someone like the Fantasy Douche back in like 2015, and then no one paid any attention to it for years (rightly), and now we have some new person discovering it, except it’s been so long it feels novel again. I was never huge in social media, but I did used to run around the Twitter streets some in my early years, because it seemed like a necessary element to the job — to growing my voice and my brand and whatever you’d call, “trying to get more people to pay attention to my work.” One of the tragedies of fantasy football analysis is the biggest grinders are eternally the ones trying to make names for themselves, but it doesn’t really matter how good you are if no one is listening. Consumers inevitably want some kind of track record and building some proof of concept that you’re worth listening to is vital. And that’s perfectly fair — for every up-and-coming analyst with wisdom beyond their years of verifiable content, there are five who grind just as hard but also don’t really know what direction to grind in, and often wind up chasing their tail (as I did plenty in my early years).

Anyway, I’ve never been huge on social, but I’ve engaged less these days, because the more I see, the clearer it becomes that everyone is sort of operating with a different foundation of accepted principles, not just in terms of what is known — although that’s definitely something — but also in terms of simply what is worth caring about, or listening to. How much emphasis we put on anything.

I’m getting somewhere, I swear. Remember that point in the tweet about the degree of certainty people have that something has to be in the past data? That’s something that as I sit back and contemplate what I knew when I started writing about fantasy football, what I learned in the early years, and what I’ve learned recently, I definitely believe has shifted. When I started at RotoViz, that site was being built on a few important core principles, one of the strongest being that any claim needed to be supported by evidence. It was simple enough, but when I got a chance to do some editing, it was a refrain I heard from FD often, and I started passing it out as advice myself to the writers I was working with. It didn’t necessarily have to be data; you could use logic as evidence, or even the dreaded eye test. But as FD would say, you were giving the reader an opportunity to follow you in your thoughts (or, as was implied, to reject the claim if the evidence you were citing wasn’t convincing).

At that time, about seven or eight years ago, it was an actual differentiator to do content that way. Two decades ago, in the earlier days of fantasy football content, and without any real advanced data to go on, much analysis you read was simply a person’s opinion. “I think this guy is good and is going to do better than he’s done,” essentially. So closer to one decade ago, using stats and data to make arguments was something that could be done in a way that might actually uncover things that the market was missing.

Today, that framework of needing to support any claim with evidence is table stakes. You can’t just go on Twitter and try to convince everyone you know ball best; that your ability to evaluate players by some magical, unproven criteria is going to be successful. You gotta come with it. You gotta bring a case. You gotta give people a reason to follow you in the claim you are making, or no one is going to listen. That’s the world the whole newer-age group of fantasy analysts came up in, and they are very aware of it. And I’ve said before, but the whole world has gotten far more comfortable with data-based decision-making in all walks of life over the past decade. Fantasy football players playing for real money are typically people from decent means with some leisure time, for obvious reasons, and that group trends toward “real-life” jobs where their industry has been infiltrated by data and numbers and evidence. It’s human evolution, and much has been made about how technology speeds up human evolution, and I’d argue our familiarity with data — as a society — is one of the clearest ways that’s true.

And if you’re following me on these claims I’m making, note that the logical endpoint is this only continues, perhaps even accelerating. That’s sort of where I think we’re at, particularly over the past year or two. I have gotten in a lot of disagreements over the past, say, 12 or 24 months, and specifically with people whose opinions I respect, where the issue isn’t really the topic at hand, but more about our underlying foundations as analysts. What we believe to be true and what we are willing to admit we don’t know. And as data becomes deeper and more engrained, it becomes more challenging to parse the little disagreements in takes between a player or couple of players. Because if you do the work — and I’ve been blessed to have some great friends who are willing to really peel back the layers of the onion to find where we’re we aren’t seeing eye to eye on our interpretation of something we should probably agree with (and both sort of expect we would) — you often find, again, it’s something much deeper and not even specific to that player/topic of the day, but rather how much weight you each put on some ancillary piece of evidence.

We know that bias is prevalent in football analysis. There are probably biases that help explain all this stuff I’m talking about, but every time I get going on something like this and wonder that, I wind up at this Wikipedia page that’s just a list of cognitive biases, and I mean as an aside, my god we are not in control of ourselves. I mean we are prone to so, so many things that impact how we think. We’re all idiots trying to convince each other we’re less stupid than the next person while hiding the obvious limitations we have. This is one of those lessons we have to learn a million times — I know fans of Elon Musk, for example, who have had to acknowledge this whole Twitter fiasco has shifted their opinion of him. I love to reference Taleb, and the concepts he addresses in his writing are so profound, and then some of his tweets make you question how you could ever believe anything this guy has to say is worth a penny. Any others we think of as geniuses probably have just done a better job of concealing their flaws and idiocies.

Anyway, it used to be easier to disagree with people in the fantasy space, because so often it was as simple as, “I’m looking at this in an evidence-based way, and you’re making a call.” That didn’t even necessarily identify that one idea was right and the other wrong — naturally, if you were evidence-based, you believed in that, and I’d argue the shift in the landscape/market indicates that using more evidence was the right call at that time, but that’s not really the point in any micro discussion. At that time, it was as simple as knowing that you just weren’t going to agree on that type of player. It simplified things!

These days, it’s more complicated. As evidence-based research is more ubiquitous, the differences in interpretations aren’t rooted in such large foundational disagreements. But those differences in interpretations are still there! This is really the whole point I want to make here today. It’s not so much what I wrote in the tweet about how everyone thinks something has to be in the data, as much as how some seem to believe the way they are interpreting the data is the only true and right answer.

And that now becomes an edge, because as I can tell you from first-hand experience, there are evidence-based cases I have believed to be foolproof, and I have watched them fail. Chaos is the rule. You cannot ever be overconfident in football data. And one of the things that’s happening with the situations that aren’t panning out over the past couple years, is some in this newer-age analyst group love to retrofit elements of the analysis to describe why a past data-based argument failed. To put that more cleanly, it’s something like, “The argument that Stat X said Player Y was a buy was so flawed because [insert sub-element of Stat X that is likely too specific to be useful for anything predictive, but does comfortably give a descriptive explain to Player Y’s failure].” These hindsight arguments always miss very obvious counterexamples, many of which you may have to go back a few years, because our data sets are not massive. But this is the kind of thing I was referencing above when I talked about analysis that was debunked years ago and then recirculates; I will just say that I see these types of logical arguments, and without wanting to call people out directly, this is where I feel like my “experience” (read: old age) plays in my favor, because I can very quickly identify rebuttals that make it easy for me to make that personal decision not to follow that person in their claim, because I don’t agree with their evidence.

And just to wrap that point up, it’s very likely the case that we don’t know why some players fail. It could be personal. Calvin Ridley’s “A Letter to the Game” over at The Players’ Tribune was one of the best things I’ve read in a very long time. There were so many layers to it, with the first and most obvious being the personal, but then also for someone like me and my work it was also another massive slap in the face about humility in my analysis. In just that piece, Ridley references playing through a foot injury he claims was much worse than was known (Point A of what we don’t know about players — their physical health), a home break-in that had significant implications on his psychology (Point B — their personal emotional health), the ramifications of that home break-in on loved ones (Point C — their family responsibilities and relationships), and more. This is where I write that I am certain I have made a case about Ridley that whittled everything down to his stats, including specifically his struggles early in 2021 before leaving the team and eventually being suspended for gambling. This doesn’t even talk about how teammates and situation and scheme can influence individual players’ stats; it’s purely about off-the-field unknowns, or at least things that were unknown to us at the time (and in many cases may have stayed unknown to us; the Ridley situation is an outlier where we’re blessed to have such a candid look at his situation).

Now, just because there are things we don’t know, doesn’t mean we can’t try to improve our understanding of the probabilities of various things. But this is where I think that here in 2023, certain understandings, or certain interpretations of things, hold probably too much sway.

I’m at this point thinking about DFS, and how its quicker life cycle of weekly events has sped up this phenomenon to some degree; these days, it’s sort of known what decisions can be made in any given week, and the best of the best at analyzing the weekly landscape from a projections and points-per-dollar type “value” perspective — that cream has risen to the top and people who are serious about DFS at the very least know those opinions. But then there are people who play in contrarian ways off of that, still, because it’s necessary. I don’t really have a point here other than to say that the reliance on specific types of arguments and information has consolidated quicker in DFS than in season-long fantasy, and is further toward that data-based approach, which I’m attributing to the quicker timeline of it all.

I’m also thinking about poker, which I admittedly don’t know super well. But I do know poker has evolved to the point where certain actions are engrained and expected and it’s — ahem — table stakes to play poker seriously. There was that big cheating scandal last year that revolved around the idea that the action taken in that spot of that particular hand was not something anyone would do in a serious game of poker for big money. There was a lot of, “If you don’t know poker, it may be hard to understand how obvious of a tell of cheating this is.” (And for me, I take that at face value, whereas you still see some people saying, “Well I make those types of mistakes in my home game so it doesn’t seem that weird to me!”)

So poker is solved, in some ways. There are degrees of it that are still open to interpretation, and that’s what keeps it fun. DFS is similar, and I think a little less solved and even more open to interpretation. But the weekly parameters are known each Sunday, so it’s fairly predictable.

Season-long fantasy — and then if you want to go a step further, dynasty — is further down this spectrum toward the “less solved” side of the equation. This isn’t rocket science. But it’s again me trying to argue that where some try to apply this almost poker-esque framework to fantasy football — that in a given spot we have to act a given way, because that’s what the data says and what we’ve done in similar spots — I’m trying to argue that: 1) that type of thinking is relied on too much already, and is perhaps baked into the market too far, and 2) it’s probably not even that correct in the first place, at least since we’re not in the 2015 landscape anymore where you could get rounds of misvaluation on pricing that made it obvious. I probably wrote 1) and 2) backward there, but 2) says that edge is closer to gone, and 1) says there may even be an overcorrection down this path.

Earlier I said this concept “some seem to believe the way they are interpreting the data is the only true and right answer” is an edge, and this is more or less what I meant, as I’ve meandered my way here. I might define an edge as a point at which you can identify a mistake in the market’s logic, such that it’s exploitable. The market is saturated with the same types of information, and as always, the flaw in that isn’t immediately obvious in real time. But it will be, I think, in the years to come. And I think in some ways, the good ol’ fashioned, “I think this guy is good and is going to do better than he’s done,” is probably now an underused argument. Something a little more macro, like a willingness to say last year that the Eagles’ whole offense would just be a lot better, and to make that case not based on past data necessarily (obviously to some degree), but more so based on predicting how the pieces would fit together.

For as much as I may sometimes mention I don’t know much, I think it’s probably pretty clear I’m highly confident in my ability to analyze football. In spite of that confidence — or perhaps because of it — I am frequently questioning what I know, and should know. And the main thing I’m seeing more and more, that I wanted to harp on today, is I’m running into a lot of situations where I feel like there’s just too much confidence in what we’ve seen, whether that’s small sample data, “advanced stats” that amount to splits data (which we know to be tough to trust in football as they take small samples and make them smaller), retrofitted analysis to explain some past bust and then pattern match who the next player might be along those lines, etc. Halfway through this post, I questioned sending it, thinking it was a little too niche and perhaps self-gratifying. But then I ventured onto Twitter, as I do, and I saw multiple tweets just in those few minutes that reinforced to me that there are points here very much worth making. One was a screenshot of a player’s data profile from a site that more or less just said, “This guy sucks,” and another was an overly aggressive analysis of a specific comp for a rookie (one player comp, no matter how similar on how many levels of data analysis, is never enough and should almost never be dug into deeply, because every player is unique, including how their data was compiled).

Probably at some point this offseason, I’ll try to reduce these concepts down into something a little more specific and meaningful. As of now, my journey to just start writing today went down the macro road, where I addressed many concepts I’ve been thinking of, about how we might want to formulate our opinions for 2023 fantasy football given the market as it is.

Very soon, I will do my big team-by-team TPRR breakdown. That will be data-based as I discuss every pass-catcher around the league, which I’ll tie to the early market prices. It will also be centered on pretty simplified data — TPRR and wTPRR with limited frills — because as I’ve written before, I love knowing what those things are measuring, and what they are not, and I prefer to analyze the layers of additional context on a case-by-case basis. This is a great example of stuff I’ve written about in this post, where I see a lot of commentary on TPRR and how it should be used by people I’d say I’m fairly confident have not spent as much time with the stat — and are not as adept at analyzing it — as I have (and am). But I get it — it’s frankly an easy and also sharp thing to do, to comment on a newer, “sexy” stat, with confidence, as if you know something about it that others do not. It signals a sort of expertise that fantasy analysts want to be signaling, and when it’s a newer stat, it’s difficult for most to spot the lack of expertise there, or the relative lack of usefulness in a predictive sense to some of those semantical points.

Anyway, that’s for another time. And it might not even be the next time we talk, because I never even circled back around to the D.J. Moore trade, and the Justin Fields note in Kate’s tweet (which I quote-tweeted at top), and we have a ton more coming this week as free agency is upon us. I can’t wait.