Dan Peterson over at Sports are 80 Percent Mental has a really interesting, and somewhat chilling, article about a profiling system called Achievement Metrics that aims to provide professional sports teams with risk-assessment tools for evaluating the probability of a player having off-the-field character issues:
Using only the transcripts from a player’s recent college press conferences or interviews, the company’s computer algorithms find patterns in a player’s words and phrases. Its not just a few vocabulary no-no’s that set off the alarms, but rather a pattern of selected triggers from a “hot list” of over 2000 words. So, unlike the Wonderlic IQ test that might allow for some pre-test cram sessions to increase the score, this analysis is much more intricate and based on an athlete’s words from the past. And, by using just the transcripts of speech, the tone, volume and pronunciation of the words don’t matter; simply the ideas and subconscious selection of phrasing.
Combining numerical text analysis stats such as word meanings and frequency with established psychological profiling theories, players can be categorized in dimensions such as need for power, level of self-centerdness, ability to affect destiny and many more.
Currently, the database includes an analysis of 592 NFL players’ speech patterns matched with their off-field behavior, both positive and negative, with a correlation algorithm. As much as this seems like a scene from Minority Report and the fictional “Pre-Crime” department, the accuracy of the results are impressive, according to the company website:
- Of the players in the database who have been arrested or suspended while in the NFL, the models placed 98 percent (104 out of 106) in the intermediate- or high-risk category based on their football-related speech from college.
- 89 percent (89 out of 100) of the players placed in the high-risk category have been arrested or suspended while in the NFL.
- Even more striking, only 0.13 percent (two out of 1,522) of players categorized as low-risk have been arrested or suspended during their professional careers.
While it’s a little unsettling to think that companies are developing tools this powerful based just on the text of an athlete’s interviews, numbers as dramatic as the one’s presented above are often a little too good to be true. It would also be nice to get a comparison of how much better the company’s algorithms are at making these predictions than a skilled human. The company presented at the MIT Sports analytics conference (video) and it is interesting, but also prompts questions about sample size and the strength of the relationships that they are claiming.
It is interesting to think about other potential diagnostic tools that might be developed to predict on-field performance at the next level. We already know more than enough about players’ physical attributes from the NFL combine and the various pro days and individual workouts that the athletes go through, and obviously the Wonderlic isn’t doing a very good job, but it seems possible that analysis of game data and footage could indicate whether a player has the kinds of decision making skills to succeed at the next level.
There was a lot of talk about Cam Newton’s inability to recall how to call a complex play in a session he did on ESPN with John Gruden, but nobody could seem to agree on whether or not this would be much of a hindrance in the pros (obviously Carolina didn’t).
But are there indicators that would hold predictive power? For example, all collegiate quarterbacks throw interceptions, but it would seem to hold that some interceptions are worse, less justifiable, and more symptomatic of problems that a player might have at the next level than others. The same might be true of runs for a large loss for running backs, or blown assignments on defense. How good are NFL teams or sports analytics firms at using patterns to do this type of predictive analysis, and are there tests better than the Wonderlic that might be developed to get at testing these skills?