Back in the early 2010's, there were essentially 2 schools of basketball analytics: exclusively box-score based stats (i.e. PER) and exclusively plus-minus based stats (adjusted plus-minus). For those unfamiliar with basketball analytics, the basic idea behind adjusted plus-minus (APM) is that every player is assigned a value and each player's value is calculated from the extremely complex system of equations formed from all the data. For example, if there are 3 players in a league of 2-on-2 (and no "opponent" players) and assuming the following results--1) during the 100 possessions A+B were on the court, they outscored the opponent by 10; 2) during the 100 possessions A+C were on the court, they were even with the opponent; 3) during the 100 posessions B+C were on the court, they were outscored by the opponent by 10--the 3 equations in this system are A+B=10, A+C=0, and B+C=-10 and the APM values would be A=10, B=0, and C=-10. Meanwhile, the most vocal box-score-based stat community was the Wages of Wins people (and their stat Wins Produced, aka WP). Without going into too much of the specifics, the major argument between the 2 sides was the preference for lowing bias of the APM community and the preference for lowering variance of the WP community.
Basically, there is a bias-variance tradeoff in supervised learning (which is a subset of machine learning with the goal of predicting an output variable from a set of input variables), and APM was essentially on one end of the spectrum while WP was on the other. In layman's terms, reducing bias means reducing the number of variables not being measured by the model (or in the case of this discussion, reducing the number of skills and thus "categories" of players that the model misvalues) while reducing variance means reducing the random noise being measured by the model (maximizing the consistency of the metric over time). Ultimately, trying to minimize either bias or variance specifically isn't the goal of any all-in-one metric; the goal is to minimize the prediction error (though there can be value in statistics intended to speicfically describe what happened, with an understanding that they're measuring some amount of luck that won't presist; for example, clutch stats are very descriptive since for the most part, luck is a larger factor than skill). When the debate between the two communities first started, both sides were too extreme on opposite sides of the tradeoff spectrum, in part because no one had publicly identified any methods for finding a compromise that lowered overall prediction error. However, the APM community was very cognizant of the flaws in their methods while the WP community was blissfully ignorant of the fact that reducing variance doesn't mean reducing error. This was evident as RAPM (regularized adjusted plus-minus) was introduced as a means of reducing the variance captured by APM and it completely blew WP out in terms of reducing prediction error (though, if I recall correctly, APM already beat WP), but the WP community still held on to their misguided virtue of minimizing variance. "Our stat is very good at predicting our stat in future years!" (Who cares that it doesn't predict who'll win the game...). For those interested in learning more about how optimizing for variance is arbitrary (hey look a Russell Westbrook problem) and can be a misguided goal especially at the extremes, Ben Taylor actually wrote a very informative and comprehensive look at various all-in-one NBA metrics.
My best explanation for the insistence of the WP crowd even in light of clearly contrary evidence is the illusion of control and this ties in the Russell Westbrook problem. Humans are wired to desire control, even at the expense of suboptimal decisions. Why do some people with high incomes/income potential (such as me 10 years ago) try to save every last penny when purchasing items from the grocery store but refuse to spend that time learning how to invest the money they make? Because the former is significantly easier and we receive consistent feedback that we're helping achieve our goal of maximizing net worth. It's psychologically difficult to stick with learning about investing when we don't know for sure we're doing it correctly (assuming we have a short stretch of outperforming the market) or, even worse, we receive information that we might be doing it wrong. Why do most people who care about global warming (again, such as me) likely overobsess about recycling, at the expense of spending time/resources that might actually make a much more significant impact than the marginal plastic container? Because it gives us a sense that we're contributing, whereas it's not clear what these other activities we could do to help the planet are.
It's worth pointing out that even the least biased metrics are still very biased, simply because they're based off conditions that exhisted in the past. In the NBA, a post player pre-2000 likely drove very efficient offense (for his time) simply because the alternative was a perimeter player who took 20-footers, so low-bias metrics based off that data ended up undervaluing perimeter creators post-2010 because all of a sudden those players started shooting 3's. This ties in a point Nassim Taleb makes in The Black Swan; when a black swan event(an outlier event that wasn't predicted) occurs, people immediately prioritize preventing that exact black swan event from happening again, as they are under the illusion that THIS TIME they're reducing left-tailed outcomes. In reality, they're being given a false sense of security by their narrative fitting that is exactly what will lead to future black swan events. As someone who used to work in Risk Management for a bank, I can tell you that risk management is specifically about reducing arbitrary risk metrics during "normal times" that offers no benefits during actual risk events. So even a stat like APM (or any of the various improvements on it) is still biased, but given that the people who value it are more comfortable with variance and less likely to be under the illusion of control, they're likely aware of these flaws. On that note, maybe it's unsurprising that we can't say the same about the founder of WP, Dave Berri; he is, after all, an economist.
Edit: hearing the news about the J&J vaccine pause and what a common to it has been (including by me, as I've made sure to get a different one). People are more afraid of the dangers they can identify and explain than the ones that are more common but they can't. People are more afraid of flying in airplanes than eating a bunch of sugar all day. Seems like an extension of the illusion of control
No comments:
Post a Comment