On small samples, the eye test, stats and narratives

November 6, 2015, by

It has been a strange start to the young season for the New York Rangers. It has been wildly varied in fan and media attitudes toward the talent level, personnel and performance of the team. I have been thinking more and more recently about the intersection between many of these concepts, and I’m going to try to keep my thoughts as organized as possible, so they don’t devolve into a jumbled mess.

I’m going to start with the game against the Capitals on Tuesday night. Both the eye test and the advanced statistics tell us that the Rangers were more or less dominated for large stretches of that game. The tremendous play of Henrik Lundqvist and the relatively poor play of Braden Holtby was largely the difference. Back in the old days, a win was a win. The Blueshirts picked up two points in the standings, besting one of their fiercest rivals in the process. They improved their season record to 8-2-2, which we all would have signed up for twelve games in. Yet, the shouting the following morning on various social media outlets was anything but positive…

This has been something of a theme so far in New York’s season. Issues have been raised (and rightfully so) about the performance of the defense. Similar concerns have been voiced about the relative lack of production from Rick Nash and Chris Kreider. The emergence of some of the depth players; Oscar Lindberg, Kevin Hayes and the fourth line, in general, have masked some of these issues. Henrik Lundqvist’s early season brilliance has similarly hidden some warts.

While the ultimate reliability of advanced statistics and possession metrics as indicators of future success continues to be debated by advocates of the old and new school, early season noise is one of the most treacherous by-products of the argument. The trump card of the week is the Maple Leafs being one of the league’s top possession teams in the early going, despite having one of its worst records. Evaluation contains many different methods and timeframes, depending on the tools for analysis. I’m not saying that the Leafs cannot be a statistical outlier, like the Kings last season, but I don’t think measuring possession samples in twelve games increments are behind the intention of the formula.

It is an accepted statistical principle that reliability will mature and stabilize over a large enough sample size. As advanced hockey statistics are only a few years old, I don’t necessarily think we have established those stabilization parameters with any real reliability, just yet. Not to mention that they are context-neutral and not intended to be comprehensive measures of complete on-ice performance. To try and argue that the stats tell us everything- or nothing, especially at this stage of the season is a fool’s errand.

So, if not statistics or rate stats, what do we base our small sample analyses on? The easy answer is the “eye test”, but that opens a whole different can of worms. This term is referred to generally by simply what you see when you watch a game. However, who said all eye tests are created equally? Wouldn’t the eye test of someone who has years of scouting and coaching experience be objectively superior to that of a habitual highlight watcher? What capacity for analysis lies within an individual “eye test”?

Another vague term that I find exceedingly problematic is “intangibles”. This term is often bandied about by the anti-stats crowd to assert the fallacy of non-testable hypothesis. Basically, if an individual’s assessment of a decision or a player’s ability does not match the stat line, it must be due to some intangible quality that imputes value that the statistics cannot find. I think the definition of “intangible” has gotten skewed a bit during this debate. Often cited as “grit”, “chemistry”, “leadership” or “standing up for teammates”, this concept seems to target specific things it would never be possible for statistics to quantify, not the ones that statistics haven’t been able to quantify, yet. For many of these concepts, the eye test will likely always be necessary.

Hockey is a game full of current “intangibles”. Broad strokes possession metrics don’t account for many things. A single fore-checker buying time for his tired line mates to change, a nice stick lift on the back-check, a smart play to lift the puck up off the boards to relieve pressure, driving hard to the net, etc., are not readily ascertainable in this context. That doesn’t mean the stats are useless, it means they are incomplete. These are the types of things that need to be quantified, not how tough a player is.

Additionally, we often find ourselves caught between the concepts of process and results. From a purely results-driven standpoint, Rick Nash is being paid more than any other forward on the roster, so he should have the most goals. Is this really the most intelligent way to look at the situation? We look at the process, as well. Rick Nash is generating scoring chances, has shots on goal, drives possession, but has a 3% shooting percentage. Is this because he is “unclutch”? Is it because he is unlucky? Is he guaranteed to regress back to his career averages? The truth is, there is no way to know for certain. People want to use statistics, as simple or complicated as those stats may be, to prove their position in the argument with absolute certainty. Unfortunately, that just isn’t possible at this stage. There are far too many variables that could be intelligently asserted to complicate the situation. Don’t even get me started on evaluating goalies…

The media’s role in the sport does not particularly help matters, either. Their content is generated to appeal to a large swath of both die-hard and casual sports fans. It is enough to keep you educated around the water cooler as a casual fan and demonstrative enough to the die-hards to keep you calling into radio shows and losing your mind in the comments sections. This week’s whipping boy is the guy who scores the overtime winner with the pun-filled headline on the back page next week. Sports journalism is very similar to film casting, some actors are heroes and some actors are villains.

So, what does this all mean? Truth be told, I’m not 100% sure. Maybe I’m just suggesting you don’t jump down a player’s throat because only one of the several forms of available analysis doesn’t like what it sees. Maybe I’m saying you should take a more comprehensive view of the situation before you start sounding like Larry Brooks. Maybe I would just like to see analysis of this beautiful game be spared the bile and ignorance of other sports coverage, and see some respect for the incredibly intelligent people who are seeking to quantify this incredibly complicated game down to a level that is digestible. This isn’t a shot at the anti-stats crowd, either. If you don’t like advanced statistics, fine. Don’t use them. Watch the game and interpret performance, as you will. Just understand that when arguing about what you see, you bring an incomplete measure to prove your point. I suppose what is all comes down to is intelligence. Do yourself justice and be credible in your positions. If you care enough to voice your opinion, care enough to learn about what goes into that opinion. Don’t pollute a website with ignorant ramblings. As hockey fans, we can be better. As hockey fans, we should be better.

"On small samples, the eye test, stats and narratives", 5 out of 5 based on 16 ratings.
Categories : Analysis, Rants


  1. Alec says:

    Just watched Colorado/Arizona last night, hoo boy what a stinker. Varlamov gave up screened goals easy, really late dropping to cover the bottom half of the net; doesn’t do a good job fighting through traffic.

    For the Coyotes, the bottom 6 were dominating most of the night, while Mike Smith was bailing the team out.

  2. Walt says:

    Interesting article, some wonderful points, and some good areas for discussion. As for tonight’s game, it can be a trap game if we are not careful. We haven’t played for several days, Nash out, new lines re-formed, road game, the Avs coming off of a stinker of a game, let’s hope we play well!!

  3. Hatrick Swayze says:

    Certainly resonated with me, Justin. Solid job as per usual.

    Question here… what is the procedure for quantifying quality of teammates and quality of competition. Often times Dave posts the bubble possession charts with zone starts on one axis and quality of teammate/competition on the other axis. Never understood how the latter is quantified.

    My guess would be that it is based off of a players corsi or relative corsi metrics. Thanks

    • Ray says:

      No, it is actually based on playing time of teammates and opponents. The presumption is that the best players are the ones most used by coaches.

      I don’t know whether this is a better idea or a worse one than your guess, but it is the way things are done.

      • Hatrick Swayze says:

        Thanks Ray, really appreciate it. Not sure I like that TOI idea but obviously that’s just a gut reaction on my part. For instance, to open up a can of worms, that would mean Dan Girardi > Keith Yandle. When in fact, he will cough pucks up at a much higher rate, yet, since he is given more ice time, the equation would suggest McDonagh is expected to have more favorable outcomes while paired with the ‘better’ aka more played Girardi. In actuality, McDonagh should have many more positive on ice events while paired with the ‘worse’ Yandle.

        Just a quick and not fully thought out example of a potential folly of using TOI

  4. Stranger Nation says:

    Is there a measure of passing effectiveness, i.e. passes attempted/passes completed (PA/PC) and passes targeted/passes controlled (PT/PC)? My guess it is ‘buried’ in possession numbers as one of the input variables but the ability to make and receive a pass is an absolutely critical part of possession hockey and scoring chance production.

    Are ‘giveaway/takeaway’ stats primarily a 1on1 accounting of possession? For example, Nash backchecks and steals the puck so he gets credit for a takeaway and the opponent gets charged with a giveaway. However, Girardi passing poorly up the boards past the NYR forward and puck is controlled by opponent’s Dman at blue line; is that (or should it be) a giveaway and does the D-man then get credit for a takeaway? Should the forward on the wall get charged with a giveaway?

    To extrapolate using the ‘passing quality stat’ within a larger sample size, the data could show over a course of 5 games Girardi has attempted 45 passes and completed 20 (15/45 = .33) while McD has attempted 60 can completed 35 (35/60 = .58). These can also be tracked by area of origin; D-zone/Neutral/O-zones.

    Combining the individual metric of GVAs/TKAs and the Passing Quality metrics may provide a better view on individual effectiveness relative to possession stats.

  5. Stranger Nation says:

    **in Girardi example originally had him completing 20, but changed to more realistic 15 in calculus but forgot to revise original #

  6. wayneg says:

    Very good article. It does seem to point to one major fact, the statistics and other factors like grit, chemistry, luck are all used to justify one’s opinion and the point they are trying to make- statistics are nothing but an way to evaluate the past and HOPE to provide insight to the future. Hockey is a sport played on a very unreliable surface, by humans with all sorts of personal motivation/demotivation issues further affected by coaching changes. Throw in the opposing teams with all the same issues and you have a hockey game. While all is good and food for thought it’s still a game and not being able to figure out what’s going to happen keep us watching and on the edge of our seats.

    • joe from newburgh says:

      Plus, hockey is such a team game that no one player’s contributions, or lack thereof, can be viewed in isolation. Does a goalie’s great save result from a teammate’s error, or would the puck have gone into the net without said teammate’s move, or did that error occur because of something that happened 5 seconds earlier? Did player A score because of B’s great pass, or did A rescue an unexpected pass and make a great shot? Did that shot go in because A picked the high glove hand corner, or did the goalie not get his glove up in time?

  7. paulronty says: I mentioned before but was not understood that small differences between numbers, say 54% Corsi vs 52% or even larger may occur by chance and stats have to be evaluated by some sort of significance result. In the above article the author uses a regression analysis to show no rel’ship, which is a much more sophisticated way of analyzing stats. Mind you I haven’t used this type of thing in years but it was de rigeur when I was doing my PhD research. I’m not against fancy stats but it seems to me that they could benefit by having someone with advanced knowledge of stats look into it. The above is the first time I have seen any of this.

  8. Leatherneck says:

    My beef with stat people is they think it is the cure all….I say it is not. Stats to give valuable information however it equates to only 50%. I don’t take stats as statistical fact. It does however show trends that need to be evaluated to see if it is fact. If say a player has had a lingering shoulder problem and has a difficult time shooting the puck and in turn he becomes a passer, does that make him a non shooting player? No.
    Case in point is the ECF when we had 4 of 6 defenseman playing hurt.

    • Leatherneck says:

      Kevin Klein is a great example, so many wanted him traded and yet early part of the season he has been our best defenseman. I stood by him and had debates why he should remain. Stats and the cap were used to justify him being traded.

    • Hatrick Swayze says:

      Leather, you’re first sentence is such a blanket statement and while true in some cases, I’d think hardly the majority.

      Stats, to me and to many, exist to supplement our understanding of the game. To attempt to quantify or explain what has transpired on the ice.

      It’s just like traditional stats don’t tell us everything. Probably more than half of the GMs in the league (maybe 2/3rds) would build a team around Toews > Crosby. Sid 639GP 304G 555A 859P. Toews 578GP 228G 287A 515P. According to stats, it’s not a comparison. According to actuality, it’s a hell of a discussion point.

      Advanced stats are the same way. Anyone who says they’re the end all be all (and I don’t think many do) are completely whacky.

      I’ll add that there is a huge fantasy contingent in sports and if I’m targeting a player, I’ll certainly look at his PDO, for instance, to see if he might be a good candidate for regression. AKA, a good buy low target. On the other hand, I may offer up Dale Weisse for a trade, cause there is no way he’s gonna eclipse 50 goals this year, right? Normally stats can help guide us with these predictions, but isolated cases provide ample opportunity for error.

      • Leatherneck says:

        How does a stat show up when a player throws a tremendous body check and wakes up his team to a win?
        Stats can never tell you everything.

    • Jon says:

      4 of 6 playing hurt and 1 (Klein) not really playing at all. He was horrendous in the playoffs. The only defenseman I thought played well in the ECF was Dan Boyle. No one else could carry the puck up ice without turning it over.

  9. Mythdoc says:

    Another point to consider: a versatile first line or two that may not score as much but will neutralize the other team’s top scorers works fine on a team that has four complete lines. That, I submit, is the Rangers team we are looking at this year.

  10. Ray says:

    I am happy with all of the opinions. I just think things that are said shouldn’t be taken too seriously. We watch the game or look at stats and we are inclined toward conclusions. We share those conclusions with others. That doesn’t mean that we should believe everything we say.

    Somebody like Larry Brooks is in a tricky position because what he really KNOWS is quite limited (like all of us), but he is expected to say stuff. I actually like him because – if circumstances change – what he says a year from now will be different from what he says now.

    One important thing to remember is that there is a huge difference between having an opinion and being certain of it — and actions can’t be based on whims. It would have been a reasonable speculation at the start of the season that neither Etem nor McIlrath would ever amount to anything. It might even be that Gorton expects both players to fail. Nonetheless, even if they only have a 20% chance to succeed, you don’t just lose one or both to waivers unless the alternative is worse.

    Personally, I expected Talbot to be better than Hank this year, but as Ranger GM, I would never have tried to keep Talbot and move Hank. If the “right” things looks idiotic, you don’t do it.

    Oh – and on small sample size, in Edmonton, after 8 games, I saw a column praising Oiler goaltending and picking Talbot as the second most valuable Oiler (after McDavid). After 13 games (same writer), is it too early to panic about our goaltending?

  11. Eddie!Eddie!Eddie! says:

    Justin, just an outstanding piece of writing. Well done!

    And Ray, I agree with every word you said (well, almost every word….I most certainly didn’t think there was any way Talbot would be better than Hank, but that’s another topic for another day!).

    There are so many sources now to expand one’s knowledge of the game we love. This site has opened my eyes to things I have never paid a lot of attention to. The advanced stats are fascinating. But like any statistical analysis, they only tell you part of the story. Joe Torre, in his book “The Yankee Years”, chided Brian Cashman, reminding him that “the game has a heartbeat” that goes beyond the stat sheet. While that’s true in baseball, to me, no sport is more of a “heartbeat” game than hockey. There are so many intangibles and variables that can influence the outcome of the final score, let alone the advanced stats. The stats obviously matter and one would be a fool to not look at them. But I’m mich more of an eye test guy. I’d bet most coaches are 75% eye test/feel in their approach, 25% advanced stats. GMs might be a little more 50/50 IMO.

    My all time favorite sports movie is “Miracle” (not exactly a shock….what hockey fan doesn’t love that movie!?). Remember the scene early in the movie when Craig Patrick’s questioning some of Herb Brooks’ roster decisions? Patrick said, “you’re leaving off some of the best players”. And Brooks said, “I’m not looking for the best players, Craig. I’m looking for the right ones.”

    That, to me, is the essence of hockey. In baseball, you can add a piece to your team, let’s say like a Reggie Jackson. A great player, but a controversial one that most of his teammates didn’t care for (if not downright hated). But it didn’t matter, because in baseball, it’s mostly an individual one on one contest. Batter vs pitcher. Your teammates can’t help you at the plate, and on defense they can only help to a certain point. So whether you have a chemistry with your teammate (other than perhaps a double play combo or a pitcher/catcher relationship), is largely irrelevant. But in hockey, chemistry is essential. The best teams are like a band of brothers. So establishing the mix, which I like to call the “people stew”, is absolutely essential to that success.

    Most of us scratched our heads last year when AV played Glass. This year, many question why Boyle is playing at all. Larry Brooks says Yandle is struggling. Others have been critical of Girardi. On it goes. The media (which I’m a part of), has an obligation to get it right. But when it comes to opinions (and that’s what they are in many cases, opinions), facts are not necessary. Because if you are going to get people to watch or listen your sports show, buy your paper, read your blog, what have you, you have to be willing to push the envelope and maybe make an outrageous claim (not based on fact…just expressing one’s view) in order to be relevant. Last year, the Rangers had the best record in the league despite playing apparently the worst 12F in history and perhaps the worst defenseman in the league. So were those two REALLY that bad? Was AV really that crazy? Or did he evaluate his options, gauge the chemistry element, and realize that this mix gave him the best chance to get two points each night? And ultimately, doesn’t the record vindicate his decisions?

    I love reading these posts. I learn so much from probably 95% of you. It’s fun to debate when we disagree. And let’s face it, if the Rangers actually were terrible like they were ten years ago, we wouldn’t have to stretch to find what was wrong. With Rangers winning the way they have under AV, it’s harder to find issues to even criticize. Nobody wants to read Brooks and company if all they would say is…Rangers continue to win…everything is wonderful. Ho Hum. And BSB wouldn’t be a very interesting place if we had nothing controversial to say, even during this unprecedented run of Rangers sustained success.

    Maybe it’s just me, but I typically don’t overthink these games. I expect one thing from my head coach–that he wins games. If he’s not doing that, I’m gong to question everything. If he is doing that, he can have Moe, Larry and Curly as his 4th line and a 3rd defensive pair of Abbott and Costello for all I care. If that’s what he thinks he needs in order to win, then sign me up! Mostly, I’m just enjoying the ride this team has been on since AV came on board (and in fairness, Torts to a large extent as well). The ride is so much more fun than it was a decade ago, when it was downright painful to watch. Now, we are perennial Cup contenders. And if we can stay injury free, and catch a few breaks, we have the roster and the coach to bring the Cup home where it belongs.

    Remember, AV’s job is not to necessarily look for the “best” players. It’s to look for the “right” players.

    • Ray says:

      Eddie, good comments though I don’t think the Rangers had worst defenseman in the league last year by a long shot. There are lots of guys that shouldn’t be trusted at all.

      I might say one quick thing about goaltending and it dovetails with your points I think. Two statements:

      This year, A is a better goaltender than B.
      This year, X is a much much better goaltender than Y.

      If you told me AB and XY both referred to Lundqvist and Talbot in alternate scenarios, we might disagree about who is A and who is B, but no reasonable person IMO would question who was X and who was Y. When building your team, if you have a choice between A and B and guess wrong, you can still win the Cup. If your choice is between X and Y and you pick wrong, you simply don’t win the Cup.

      The right players are the guys who don’t keep the rest of the team from winning the Cup.

    • Paulronty says:

      Eddie, I was trying to give you a thumbs up but hit the wrong button. Great post but Abbott and Costello on D would be a lot of laughs.