Superhero Metrics, Part IV: Disadvantages
The WIMCS is not without its own biases and shortcomings. One of the biggest flaws I can see is that it measures only current popularity, without regard to historical popularity or influence.
For example, I’d guess that Iron Man only comes in at such a high ranking because of the success of the 2008 and 2010 films. (Indeed, the page for the 2008 film has a slightly higher WIMCS ranking than the page for Iron Man, himself, suggesting that the film is driving interest in the character, rather than the other way around.)
In the case of Iron Man, specifically, the films were released after Wikipedia came into existence, which means that we can look back in time at the page history to see how the release of the films affected its WIMCS ranking.
However, in the case of such cultural touchstones as Superman, the institutional memory of Wikipedia is far too short to remember a time when Superman was the biggest superhero by far, instead of tied for second with Spider-Man. (Still, if you’ve got an article about you in the Old English Wikipedia, there has to be a sense in which you’ve culturally “arrived.”*)
Another big flaw in this methodology is testability. The WIMCS was developed as a more precise alternative to keyword searches, but in order to verify its accuracy, it needs to be checked against some other metric, which brings us back to keyword searches or some other technique.
For the moment, the only metric I’ve tested it against is my own gut reaction. So, if Bizarro had a higher WIMCS ranking than Superman, I’d know the methodology was flawed, because I know that Superman is more culturally significant that Bizarro. However, if Superman had been slightly ahead of or behind Spider-Man, I don’t think that I would have been surprised. (Indeed WIMCS rankings may not be fine-tuned enough for it to matter if one article ranks slightly ahead of or behind another one, although I still believe that large differences in rankings should be considered significant.)
Yet another disadvantage is that this technique can measure a topic’s overall popularity, but not its popularity within a certain class. So, Beethoven and Bizet are both members of the class “Opera composers,” and Beethoven’s WIMCS rating is 2.5 times that of Bizet’s, so Beethoven is the more popular opera composer, right? Well, no. Although Beethoven did write one opera (Fidelio), he’s much better known for his musical compositions in other forms, while Bizet’s Carmen is a staple of opera companies. (And, indeed, the WIMCS ratings for Fidelio and Carmen bear out this distinction.)
And the last disadvantage is fairly straightforward: This technique doesn’t work for topics that don’t have their own Wikipedia page.
In part V, I’ll discuss potential applications for this ranking technique.
2 Comments:
.
Stats and rankings have been on my brain of late. Thank you for feeding me more.
Happy to oblige!
Post a Comment
<< Home