Stealing Signals, Week 14, Part 1

Analytical sample sizes, plus TNF and the early games

There’s always a hindsight element to the way people are thinking right now, that implies we should have all seen it coming. I wanted to write about Saquon Barkley in this intro anyway, and what I think about him for 2025, but he makes a great starting point, in that he’s being widely understood as an industry-wide miss, where fantasy analysts en masse were out on him — obviously not literally everyone, but lots, as the narrative has been told on social — and the little guy was right.

And I mean that’s fine, as far as a narrative; I’m certainly not in the business of defending all the other analysts in my industry, which has been something of a sticking point for my career that I think I wind up coming across pretty critical to people whose support I should probably be trying to win. But I do think I’m pretty tied to the truth, or what I think is true, so don’t think this is me blindly defending fantasy analysts when I say that the whole narrative there doesn’t add up.

The question I’d ask for something like this is, “How large is the sample?” Of course, in a sample size of one, the little guy wins the Saquon situation. If that’s the extent of the point, sure. I was actually thinking about D.J. Moore a little bit ago, and how most analysts I knew felt like his best ball price never corrected after the Keenan Allen acquisition, and how the whole Bears’ situation has been tough but even as it has picked up lately, it’s clear that Moore was too expensive relative to his teammates. The point of this part of the intro is going to boil down to me arguing you won’t hear anyone saying, “The fantasy community was all correct on D.J. Moore.” And that’s because we ignore the part of the sample that doesn’t fit the narrative, which from an analytical perspective is where you get into trouble.

People are absolutely going to go into 2025 with renewed vigor to do the opposite of whatever the popular industry-wide take is. If fantasy analysts all seem to agree on something, there will be, “He’s this year’s Saquon” comments, and that will carry with it the connotation that the fantasy industry is missing something, and even imply that it happens with some regularity. Interestingly, I don’t even think there’s very much consensus from the whole industry on many more players — I’m not sure I can think of how to make this sample size point here because I can’t get much further with the sample size — but the idea does stand.

One way I was thinking about it was how the discussion last week was all about how this version of the 49ers failed. I never said it anywhere in content, but I felt like that was a little bit premature. They’ve been hit with a lot of injuries, but I still didn’t think this team was doomed. They have a ton of talent. They could absolutely still get hot and win out, and then make noise in the playoffs. I think after Sunday’s dominant win over the Bears, you’d find some more people willing to hear that idea, but this is still a 6-7 team two games out of the playoffs with four to play. They might actually be done, and it might not even be in their control.

My point with them would be the Bills were in a similar spot in 2023, and people were saying similar stuff, and then they won five straight to close the year and make the playoffs, and then won a game, and then lost to the eventual champions by 3 points where they missed a kick inside two minutes that could have tied the game.

So how big is the sample size for a team like the 49ers? People love to say it’s massive, and say things like, “Of all teams 5-7, only 3.14159% made the playoffs.” But the reason we care so much about the 49ers specifically being 5-7 is we never expected them to be 5-7; we’re inherently calling their true talent and expected win percentage much higher than even .500. That was the same deal with last year’s Bills, and I’d suggest that if you made a group of just those types of teams, with some type of intellectual honesty about that sample, there’s some potential for the type of occurrence we saw with the Bills just last year. A good team starting to play well. I understand the odds are stacked against them, but if I could bet on any team to go 5-0 the rest of the way, especially because they need to, the 49ers would be somewhere near the top of my list (they still host the Lions, though, and that’s a really tough game).

I’ve now created two very difficult-to-define cohorts, the first one being positions an entire industry held with something close to unanimity, and the second being really good teams that didn’t have W-L records to support it, and I don’t have much interest in going further down either rabbit hole. Another that crossed my mind is the rookie chasing in fantasy, where I still hear about Laviska Shenault and Skyy Moore — and hell throw in N’Keal Harry and Treylon Burks — but those critics never acknowledge how it’s the same process that lands you on Brock Bowers and Bucky Irving and Jayden Daniels and other great WRs like A.J. Brown back in his rookie year, and on and on, and how big the impact can be for those types of players. That’s an easier one for me to be like, “You’re going to focus on the misses, but the sample is larger, and the results are pretty clear that the hits are impactful enough to more than offset the misses.”

So when I talk about “How big is the sample size?” what I’m thinking is there are often examples of the thing that’s being pointed to that directly refute it, but we have this strong bias toward the high-profile miss, or whatever’s most recent, or what have you. There always seems to be this hindsight that we should have seen it coming for some reason.

If that reason is a new piece of analysis that can help us going forward, it can be interesting. But in most cases, the reason just creates a sample where there are obvious counterexamples being ignored. The bias toward the high-profile miss often ignores the data points that specifically went the expected way. If I tell you a bunch of rookies are going to smash, and they do, it’s easy to think they were inevitable. If I tell you the Bills won all those games and went and played the Chiefs tough in the playoffs, they don’t matter the next time we talk about a similar team that way, because the Bills are just that good. Except also, the 49ers are, too, is what I’m saying. If you’re telling me to predict 2025 records right now, I’m probably telling you the 49ers are going to have a season next year that rivals how the Bills came out this year and currently sit at 10-3, despite, again, being in a position where they were 6-6 last year and people thought the era was over.

It wasn’t inevitable for them to bounce back, either, and they’ve done a ton right in adjusting the way they do things, and the offseason moves, etc. My point is simply that people don’t see the whole picture well.

Here’s one last example, to this half-baked point. I’m often criticizing film watchers, but I’ve also argued more and more strongly that people need to watch all the games to really understand what’s going on. One of the things I think I see with film watchers is what I’m trying to say here — they don’t look at the full body of work. The sample is already not large enough, but they will watch some select plays, look for traits, and then make a declaration that the film says something about a player.

The problem is the full body of work is massive. I’ve talked about your best 20% of your plays versus your worst 20%, and I remember recently talking about how I saw some positive film stuff on Drew Lock, and it just tied in so well to me where yeah, if Drew Lock could just cut down on the inaccurate throws and turnovers, the rest of his sample does look good. Same with Will Levis, right? “There are good things there,” you always hear. But that’s what makes the great ones great is they make good plays without constantly slamming their fingers in the doorway. You also have QBs who are too conservative, who don’t make mistakes, and people don’t love them, but again the whole body of work is the issue, right? It’s not impossible to avoid mistakes if you’re constantly checking down, just like it’s not impossible to make some correct decisions and some big throws if you stand in there with confidence and sling it with reckless abandon and are willing to throw three picks a game like Jameis Winston. But we can’t judge people by slicing out some percent of their plays and saying, “If only they didn’t do these things” when they refer to the tradeoffs of the good things that player brings. And that’s what a ton of film-watchers do, is highlight these good plays and not really understand that the whole sample of a risk-taking QB — for example — is super relevant, because taking risks leads to good and also bad plays.

In all of these examples I’m trying to give, people try to formulate opinions, or analyze outcomes, without being honest about how large the sample size really is, and what’s actually helpful information for the next prediction. And then, because they want to fit a narrative rather than understand what they are trying to figure out, they make a bad prediction the next time.

And this specific idea that when an expected outcome occurs — like when the entire fantasy industry is correct on a player — it is far more likely to be ignored, is super relevant. There’s never any love for hitting expectation, which then amounts to an expectation of never being wrong, and then an overemphasis on outliers. My honest opinion is this idea the whole industry missing on Saquon Barkley is a reason to fade the next industry consensus would be not just wrong, but absurd.

Each situation is unique, and you’re talking about some hypothetical future consensus among a bunch of people looking at a lot of different information, but definitely being informed. Telling yourself that a consensus among the most informed people in an industry is a signal to do the opposite because you’re sure the industry “is always wrong” in these types of situations is crazy. You guys don’t need me to tell you that, because you’re a pretty well-informed audience, but I was just thinking about it and laughing.

But it also got me on this larger point, about how people always want to tailor their reasonings in ways that ignore obvious counters that would also be a part of the sample. There’s often this idea that something “always” leads to something, and most of the time it’s like a sample size of two similar things someone is pattern matching, and I can think of two other examples that could also be tied in and we’d have a 50/50 sample of two hits and two misses. And that’s a concept that’s always worth keeping in mind.

---

Let’s talk about Saquon specifically, and why this was the week that I bought into drafting him in the top five next year.

For those of you who have been in on Saquon, you could fairly and justifiably criticize my little journey with him of late as way behind the times. So why am I doing this now? Because I do always try to be honest in my writing, and obviously I’ve been skeptical about him since the offseason. I couldn’t just flip that. It’s taken me longer, full stop.

But this game was fascinating, and it forced me to buy in, in a way I don’t want to forget by next August. No receptions. No TDs. A long run of “only” 18 yards.

That’s the profile of a TRAP back, where without High-Value Touches, and unless you can hit for extreme efficiency with long runs, you’re just not supposed to be able to score. But Barkley still averaged 6.2 yards per carry, and racked up 124 rushing yards for an easy floor. He did add a 2-point conversion to get to 14.4 fantasy points, but I mostly want to ignore that and emphasize that a 12.4-point floor in a game without any real HVT production, is insane.

One of the ways I talk about HVTs is to say that with the reception element, you can see the truly elite like a prime Christian McCaffrey have something like a 15-point floor without a TD. The idea is that non-HVT RBs don’t have that kind of floor. Even the one clear outlier in this era of HVT writing is prime Derrick Henry, but even he isn’t a great comp that would get at the specific way Barkley compiles yards. Prime Henry was inevitable with the long runs, like Barkley feels, but sometimes you could cut Henry down in the backfield enough to stifle the rushing efficiency overall, and he did have some low-floor games.

This was the perfect runout for a tough outing if you were facing Barkley, and if you were watching, you felt like you escaped with him hitting in that 12-to-15-point range. Simply put, that’s the sign of a truly elite fantasy range of outcomes. He could have caught a couple passes, but the defense tried to take that away. He could have easily scored a touchdown, and got touches in close and was tackled at the 1 again. He did get the 2-point conversion. The big runs obviously could have been there, as they so often have been.

I realize some will say, “But this was against Carolina.” My point is you want a lot of ways to get there, and in this case, Carolina prevented the huge plays. No runs of 20 yards. So beating up on a bad rush defense that does take away your receiving and where you can’t get big plays and also you don’t score isn’t exactly my version of “you just got lucky.”