What Review Score Should We Give Metacritic?

By Dennis Scimeca
Saturday, July 16, 2011 at 10:00 am
What scale should we use to grade Metacritic? Letters, numbers, or stars?

Metacritic's value lies wholly outside Metascores, and wholly within its listing of web links to video game reviews. If the site is used strictly as a gateway to reading the reviews of the outlets we trust the most, Metacritic is your best friend if you're considering the purchase of a new video game.

When I check a game out on Metacritic, especially a triple-A title, I find myself scrolling through an incredibly long list of review outlets. Every so often I click through to all the reviews for a title, and find myself looking at many outlets I've never heard of, whose review copy is poorly-written, offers no insight or unique commentary, and whose critical process seems thin or nonexistent.

The question of whether Metacritic should include outlets that use unpaid, inexperienced reviewers becomes easier to answer when we acknowledge the influence that Metacritic has on the video game industry. For better or worse, Metascores can affect bonuses, publisher stock prices, and even continued employment for game developers if you think about it. Being listed on Metacritic should be a statement of legitimacy for a video game review outlet, and I currently don't think it is.

Marc Doyle.jpg
Marc Doyle

I reached out to Marc Doyle, co-founder and current Metacritic Editor-in-Chief, to ask about the preponderance of outlets I didn't recognize.

"Bear in mind that many of those publications are system-specific. So you may not have heard of the 10 - 15 iOS specific publications such as 148apps, SlideToPlay, etc. which are very popular among those groups. Many of the sites in our games slate are leading publications in non-North-American countries - it's highly unlikely that you and your English-speaking peers regularly consult Multiplayer (Italy), GameKult (France) or Gamereactor (Sweden/Denmark). We've also always maintained a place on Metacritic for a cross section of independent sites that do high quality work. So sure, we cover the IGNs, the GameSpots, and the Game Informers of the world, but we're open to including smaller sites that have consistently demonstrated quality analysis and interesting insights that we feel would be valuable to purchasers in making their decisions."

"Quality analysis" and "interesting insights" are where I raise an eyebrow. Some of the amateur, fan-site outlets listed on Metacritic seem highly sketchy in light of those pre-requisites, and even if we account for the iOS-specific sites, and focus only on the North American or British outlets, consider the following.

On its "Which critics and publications are included in your calculations?" FAQ page,
Metacritic lists 46 outlets for movie reviews, 54 outlets for music reviews, 41 outlets for television reviews...and a whopping 154 outlets for video game reviews. It's difficult, in the stark light of these numbers, not to see Metacritic's list of video game outlets as bloated at the very least.

I wonder which one weighs the most?

Metacritic weighs reviews, such that some outlets factor heavier into Metascores than others. This implies a simple truth which may be unpalatable or offensive to some, but which is no less true for it: some opinions are worth more than others.

I'd like to believe that there's something to be said for a critic, of any art form, who has to fight their way through the crowd to earn a paid position at an established outlet. When we look at the publications that Metacritic lists for books, music, and movies, we overwhelmingly see a list of acknowledged, respected journalism outlets, and that's how it should be.

There's a difference between someone who critiques games professionally for a living and has many years' experience doing so, and a video game fan who applies for a free writing gig at GamezBlogX and has little to no experience.

I have worked on both sides of this fence. There are some very talented and dedicated writers working at the amateur video game blogs I've written for in the past, but it's not the same thing as doing the gig professionally and having so much more at stake when you file a review.

No one is going to change Metacritic's impact on our industry in the short term.

Doyle takes the responsibility of curating the sites he lists seriously. "We re-assess the quality of all of our sites annually, but unless we notice that quality has depreciated or there is some other critical issue that arises (corruption, unprofessional behavior), we are not quick to drop a site that we've been tracking and that has had a history producing quality analysis." Doyle recently discussed dropping an outlet due to corruption with ComputerAndVideoGames.com.

Corruption and professionalism are baseline concerns, not metrics for identifying quality. I'm curious about the second of the two standards Doyle states as necessary precursors to being listed on Metacritic, "interesting insights," which we could boil down to "originality." Yes, many of the professional video game journalism outlets say the same things as one another because sometimes there isn't much to say about a video game, but it's not an unfair question to ask just how many amateur sites we need parroting the same sentiments as the professionals.

It's reasonable to suggest that video game review outlets which may have been appropriate to list earlier in Metacritic's history, before the site was so established and influential, may not be appropriate to list in 2011 when Metacritic has established itself so prominently as a force in the industry. Doyle is not unaware that this may be the case.

"It was definitely a different landscape in 2001 in terms of the publications which were reviewing games and the awareness of Metacritic. There were some tremendous publications in 2001 (Games Domain, Daily Radar) which no longer exist, and some great ones have sprung up since then. Some publications which were top quality in 2001 may no longer be considered as reputable as they were or as good as the body of other publications against whom they are now competing in 2011. These are things that I regularly consider." 

Vancouver Rioter.jpg
"That game sucked! I'm gonna write about it on my blog!"

Metacritic's FAQ page has this to say:

"For games and music, we work to identify publications that (1) are well-regarded in the industry and are known for quality reviews; (2) actually seem to produce quality reviews (or, if not, are so influential in the industry that they have to be included); and (3) have published a good quantity of reviews."

It's point number two, which explicitly allows for the inclusion of outlets that produce low-quality reviews but still generate a lot of web traffic (read: "influential in the industry"), that should inspire those of us in the professional video game press who want our institution to be respected as a legitimate branch of the media, to be vigilant about which outlets are listed on Metacritic.

While it's not Metacritic's job to regulate the video game review industry, Metacritic does hold much more sway and power than any individual video game journalism outlet in existence. That's why everyone wants to be listed on Metacritic, isn't it? The good news is that Marc Doyle is quite willing to engage on this issue:

"As part of my research, I'm always willing to listen to gamers, journalists, and others as to which publications they feel deserve or don't deserve a place on Metacritic, which publications they feel are most reliable / credible."

We should take him up on the offer.

First Person is a weekly column by Boston, MA-based freelancer Dennis Scimeca. He is not sure what this recurring byline ought to say, but feel free to give him a hint on Twitter @DennisScimeca.

Email Print

Join The Joystick Division!

Become part of the Joystick Division community by following us on Twitter and Liking us on Facebook.

More links from around the web!