Metacritic is something that should be extremely useful…in theory. After all, it takes reviews from all over the internet and boils them all down to a single score based on what many different sites and people are saying. It should be a good way to get a general consensus on any one game, but somehow things didn’t work out that way. Instead, we’ve ended up with a poorly-policed database scaled with nebulous rules and choked with opinions of people who haven’t even played the game. As far as its intended purpose goes: it’s a train wreck, but one that many publishers use to gauge how popular their games actually were.
Just the same, how many gamers know or care about its problems? Wanting to know how important Metacritic scores were, our own Jarret Redding asked this question of the Reddit community. I found the answers to be really heartening, showing that gamers were completely aware of the system’s problems. Out of the 240+ answers received, almost every single one spoke of using Metacritic as a tool and not taking it at face value. Almost each reply spoke of looking into where the reviews had come from or using it as a way of digging deeper into the game. Many were so disgusted with its policies and hidden rules that they didn’t bother with it at all. Others either didn’t trust reviewer or player scores to some extent for various reasons. Not one person took Metacritic’s average without looking into it. Not one person trusted the score implicitly. Luckily, many of them were even polite enough to tell us why.
Metacritic allows for players to write out their own short reviews of the game. While this is something that I found really useful back when GameFAQs started doing it, the state of Metacritic is a mess. It’s a really good idea in theory, as what do gamers have to prove to anyone? No one is going to dock their pay if they like or hate a game, so they have absolutely no reason to be biased towards a game. If you’ve ever been on this thing called the internet though, you might already know a thing or two about how unbiased people can be once they’re online.
User “O2J” was kind enough to provide us with some hard data on the subject by looking up the player review scores for Diablo 3. While I knew that a lot of people were mad about how the game was handled, I wasn’t quite prepared for the venom I saw in the scores. A full third of all player reviews scored the game as a 0, many of them citing DRM or connection errors as a reason to why the game was one of the worst things ever created by human endeavor. The connection errors really were a mess during those first few weeks of release, I’ll admit that. They were fixed shortly afterward though, so is it still fair to say that the game deserves a 0 rating for a problem that no longer exists?
A big problem here is that you have a bunch of angry people with free reign to plunk down a player review that literally states that they haven’t even gotten to play the game yet. If you’ve never managed to get into a game then that’s a big problem and it needs to be addressed. It also means that anyone can hop onto Metacritic at any point and give a game a 0 rating for no reason. They don’t have to prove they’ve played the game or even know about it. If someone so much as dislikes the title and has a group of like-minded friends, then they can easily slam Metacritic with tons of bad reviews.
What’s really stopping them from doing it? It’s not as if it’s difficult to hop on board and start flaming games left and right. Signing up is no harder than getting a new Hotmail account and takes just about as long. Once you have your account, you can go right on in and start reviewing games. All that Metacritic asks of you is that you don’t type in all capital letters, that you make your review at least 150 characters, and that you try not to misspell anything. That’s all of the criteria you need to write a review there. GameFAQs also allows for players to write their own reviews, but on that site the minimum review size is 400 words. Not only that, but the reviews are looked over by a moderator before they go up for everyone to see. It still lets players talk about why they love or hate the game, but it stops them from just popping on, spewing hate, and then scoring the game as 0. Even if you wanted to do that on GameFAQs it would take you several hours, not five minutes.
It’s far too easy for just anyone to walk onto Metacritic and start slamming on bad reviews, but what’s the big deal? Diablo 3 happened to be pretty polarizing for many of its longtime fans when it first came out, so it drew an unnatural amount of heat. Not every game is going to have a huge group of gamers who want to tear it down for no good reason. Call of Duty: Black Ops 2 might have suffered a similar fate to Diablo 3, but these are more unique cases when a high profile is hated because there are lots of people who just want to hate on it. The average game seems to score a little more in keeping with what reviewers are saying without having a disproportionate amount of low scores, so is this really a big problem?
To me, it doesn’t matter if it is a big problem or not because the idea that it can happen makes the numbers for user reviews useless. Like I’ve said, you don’t need to prove you’ve played the game or even have that much knowledgeable input in order to put in a review score. At least with a professional reviewer I can assume that you’ve been looked in on to some extent. Even if I go to GameFAQs, I know that someone has read your article over and approved it before showing it to the world. With Metacritic there is no such filter, and while it might not be a huge problem for every game that comes out, the idea that fanboys and haters can bloat a score one way or another just makes the rating meaningless.
It’s not that the critic reviews are that much safer either. User “nothis” pointed out that gamers are worried about whether the reviewers have been bought by the game companies. I spoke about this at length a few months ago, and I’m glad to see that this is a problem many gamers are aware of and taking interest in. Simply put, getting free games puts a bit of pressure on a reviewer to score the game higher than they normally would in order to keep up the good rapport with the developer or publisher that sent it their way. When you throw advertising dollars into the mix, you can get pretty extreme situations—ones that result in reviewers getting fired for hating on a game that obviously threw a lot of money into advertising on the site. Check out my article for a more in-depth look at that problem, though.
The scores can also be a source of some headache. As user “Aftershave 92” pointed out, if you look at reviews for any period of time, you end up seeing a lot of games rated somewhere from 7-10 (if the rating system is out of ten). Everything below that tends to be some bleak wilderness where the numbers could mean anything. Really, how many people are going to be all that interested in playing something that received a 6? Once you end up below those high numbers, there’s a good chance that most people won’t even bother with the game at all, so the game could really score anything below 7 and it would still have the same effect. Games cost a lot of money, so no one really wants to shell out a chunk of their paycheck on anything that’s less than pretty good.
It makes sense, but it means that there’s a whole lot of stuff floating around at the top of the pool. Instead of having all of this room to play around with, most reviewers can find themselves stuck shuffling a game around the top three or four rungs, trying desperately to decide where it should go. Did I really like this game better than this other game? Do I really consider this game to be on par with this other one I reviewed a few months ago? I don’t really like this game more than some other game I scored low before, but that would mean ranking it far down below where I think it should go? It creates a huge mess.
This is just one of the reasons why high-profile games tend to get these colossal scores on Metacritic. The system has been played with to the point where anything even remotely worth playing gets a 8, 9, or 10, and anything below that may as well be in the Old West for how much order there is. The end result of all this playing around is that the reviews don’t mean a whole lot when they’re boiled down to just a number sometimes. Unless the specific reviewer is making a conscious effort to use the entire scale of numbers, then you’re going to end up with all of these high-scoring games and no basis for any comparison. Given that this is what most of the industry has been doing for several years, it has gotten to the point that most games can score quite high just because they’re not terrible.
What does any of that mean for Metacritic though? Well, if you’re trying to provide an average scoring system for games, what good is it if most of the games are all getting similar scores? If many games are getting scores from 8-10, how do you meaningfully judge them against one another? If almost every high profile game scores a 9 or 10, does this change the meaning of those numbers? Does a 10 rating mean absolute perfection as it should, or just a really good game with minor problems? Is a 9 from this site or reviewer as good as a 9 from another site, or is it closer to an 8? Metacritic tries to solve this by making the numerical score out of 100 to give the average scores some wiggle room, but it does nothing to cure the problem with the reviewers themselves.
That calls up Metacritic’s problems with how it scores games though. Given the many different numerical scores that exist across all gaming sites, Metacritic had to come up with a single way of scoring everything. To do that, they take the score from the site that has submitted it and make it out of 100 instead. As user “Cabbage_Vendor” points out, you only have to look at a 3/5 on a site to see how that could be a problem. If I were to give a game a score like that in a given category, it’s because I felt like it did a few interesting things to make it above average, but not amazing. If you look at it as a 60%, see me after school because you only just passed your math test, then it looks abysmal. That 3/5 might not be too bad on the site it was pulled from, but put into another context…it looks like the game is just a bit better than functional at best.
In the same case, a 4/5 is a great title, but the translation to 80% on Metacritic puts it right at the bottom rung of the most common ratings for high-end games. Worse, as was seen in the case of Fallout New Vegas, you can see developers losing their bonuses if their game doesn’t achieve a certain score on Metacritic. In this case, it had to achieve an 85% or higher—something that would be just about impossible if Metacritic had pulled its scores from an unusually large number of sites that used a system that ran on a five point scale. This is a score that’s almost perfect as far as the site is concerned, but in the eyes of Metacritic, it’s just pretty good.
The change in score doesn’t reflect the reviewers’ intention or belief of the game’s quality. Changing the scale means changing where the reviewer might place the game, and as such, it’s taking the score and making it meaningless. Any change in what the reviewer put in as a score should be something that’s done by the reviewer himself and no one else. Doing it this way isn’t much better than guessing, and it defeats the whole purpose of putting all of these review scores together. Instead, you just have a bunch of meaningless numbers that don’t mean a whole lot. They give an idea of the general consensus, but there are some huge gaps in where the numbers came from.
User “mbdjd” left a fun link to Metacritic’s own FAQ, and a lot of the stuff in there is fascinating. One aspect of the system I hadn’t even thought of was if the review site that Metacritic gets its review from doesn’t include any numerical score at all. Instead of asking the reviewer to come up with something, they will assign a staff member to come up with a score. Now, this isn’t something that gets done in a vacuum according to them, but it is still a staff member that comes up with the final score. That’s even more frightening than adjusting a numerical score to fit the 100 point scale, as now we’re just guessing at what a score should be.
It doesn’t sound so bad, but just look at my own review of New Super Mario Bros U. I spewed nothing but hatred at the game for three pages, but in the end, I still had to admit that it was a decent game. It was one that should have been much better than it was, but as far as gameplay went it, was a solid game. If someone other than me had scored it, they would have probably put it a lot lower based on many of the negative comments I’d made. You could argue I should have made that clearer in the review, but I felt that the focus needed to be on how much the game borrowed and reused rather than its quality as a new game all on its own. I felt that was the most important thing about the game, but there was no way you could know about that if Metacritic had just assigned a review score. I could go get that score changed if I noticed it on Metacritic myself, but I think that’s a little backwards. It relies on me to keep an eye out for my score when they should be asking me to begin with.
Even more fun is the fact that some sites are given more weight than others. It makes sense though. A lot of high-end sites can afford to have really professional reviewers in, and their opinions are given a lot more weight with a lot more people. That being said, we’re back at the problems that people have with crooked reviewing trends in the industry. More money doesn’t necessarily equal more integrity, so some of these sites could be giving biased reviews based on things like advertising funds.
Another problem comes from these bigger sites using unproven freelancers to write reviews. If you can recall back in November Gamespot published a review for Natural Selection 2; a review that was written by someone who’d barely played the game. It was pulled by Gamespot only a few days later, but at the time of this article, is still up on Metacritic. Given that the review is just plain incorrect I would think that they would want to pull it to keep the score correct, especially given that it’s the only low critical score the game received. Yet, it’s somehow still there. Even though it’s completely wrong, there it remains for any unsuspecting gamer to read and take into account, and for it to drag the game’s overall score down.
That’s not even the best part. Since that score is from Gamespot, a high-end site, it could carry a huge weight with how the game calculates the score. It doesn’t appear to be doing much damage given the pile of positive scores, but it’s still unknown how much weight it carries. Metacritic refuses to divulge how it scales its scores or chooses which sites to give more importance too—something I thought would be extremely important. After all, this is an average, so if not everyone is carrying the same importance then it would be nice to know where the numbers come from. I can understand that some sites should carry more importance than some new no-name site that just appeared, but I’d really like to know how that is being factored in. It can make a big difference if a big-name reviewer or site likes or dislikes a game. I’d like to know who that was in case it’s an opinion coming from a site I don’t trust.
That seems to be the most important thing I’ve drawn from all this. While Metacritic has reams of problems, gamers are already aware of that and are just using it as a tool. User “Dreadgoat” goes on Metacritic with a couple of games he likes in mind, using them to find like-minded reviewers. If he can find someone whose tastes are close enough to his own then he’ll watch that reviewer’s opinion closer. In the same vein, that method will show you reviewers that don’t like what you like and allow you to dismiss their opinions. It’s a neat way of finding new reviewers.
User “grizzly_” takes a handful of negative reviews and positive reviews and uses them to get a general feel for how people are taking the game. Not to sound biased, but I do the same thing. If I really want to see how people feel about a game, I’ll read the best and worst reviews I can find because they typically talk about similar things. What someone hates about a game could be something another person loves, but reading each one will give me a good idea of what the good and bad points are.
Another good method came from user “epicgeek.” In his own words: “Nothing can really compete with a trusted internet personality doing a ‘let’s play’.” Watching someone just play the game can be the most straightforward way to decide whether a game is something you want to play or not. I’ve found dozens of awesome games by watching passionate people playing them. You get to see firsthand whether the player is having trouble with controls, whether the graphics are muddy and ugly, or whether the story is idiotic. You run the risk of a few spoilers, but you get to see what the game is like on your own terms.
The key word on this has been “trust.” Metacritic looks to build an entire network of scores, but there are too many reasons why such a large group of people can’t be trusted. At best, Metacritic is a tool that will help you zero in on a person you can trust and then either listen to them or watch what they play. It’s a convoluted thing that doesn’t deserve to have any importance placed on it, as it’s nothing but a breeding ground for chaos and misdirection. Gamers know to only use it sparingly, so I wish that the publishers and other important people in the industry would see that fact. Everyone else seems to know it but them.
Images courtesy of unwinnable.com, reddit.com, geekosystem.com, smashboards.com, thecontrolleronline.com, metacritic.com
