Editorial - The Ratings Game

We've just updated our review key here at the Review Corner so I felt like it was a good time to reflect on the ratings we give games and why we give them. Those stars mean an awful lot to some people who might pay more attention to that flat one to five star metric than they do the paragraphs of professional prose that our writers and 'casters crank out weekly, and to others they might mean next to nothing versus what is actually in the text or content of a review.

Let me level with you, reader. I do not like applying a numeric value to a game's relative quality, interest, innovativeness, or level of fun on offer. As a writer, I want you to read all of those paragraphs and decide for yourself where I fall along a general paradigm that ranges between a binary system of "good" or "bad". I'm sure our 'casters want you to be riveted to your seats throughout their episodes as well. In my past writing, I've shied away from points, stars, flaming devil heads, wiggly joysticks, screaming faces, or anything of the sort. But when I was writing at the video games site Gameshark.com, my Editor in Chief there, the great Bill Abner, told me that we've got to ratings because a) readers like it and b) we wouldn't get on Metacritic without them.

OK, I'll admit it. GamePro's old rating system was actually really good. But we can't use it.

Of course, Metacritic doesn't track board game reviews so it may seem irrelevant here. But the impact of sites like Metacritic, Rotten Tomatoes, and any number of sites that apply a numerical rating have created an expectation. Readers want that ultra-high level, eyeball read on whether a game is good or not. It's a "grade". But this is a skewed concept, because the numbers at which "good" ends and "bad" starts are very, very hazy. And I would argue that Siskel and Ebert had the right of it- thumbs up or thumbs down, with the opinion residing in the discussion.

On Metacritic, for example, if you look at a review with a green color-coded score of over 80, you get a sense right there without reading any of the aggregated opinions that it's something pretty good overall- without ever having to read anything, of course. But once you hit 70, that color changes to yellow. Even though a 70 is on the better side of 50 by two points, we're starting to get into a "caution" zone. Once you get below 40 and the number turns red...well, it doesn't really matter if it's a 20 or a 30 because the main takeaway is that whatever it is, it's not good. Red light!

In reality, everything below an 80 may as well read "At least as bad as Sucker Punch".

In the hobby gaming world, this screwy logic occurs too regardless of the convenient color-coded, and in fact myself and some of our writers have had discussions about this very topic- what is a 2.5 star game, and what is a 3 star game? Where is the line between "good" and "bad"? If I read a writer's review and feel that they are writing a 3 star review but they've rated it 2.5, then where is the line of demarcation? I have had reviews across my desk with the exact same star rating, but wildly different levels of what is "good" in them. It comes down to an editorial judgment call rather than an empirical measurement.

It's tricky business, and this is why our review key is an important guide. I wanted our readers, viewers and listeners to get a sense of what the guidelines are, and more importantly with the expansion of this key to get a better sense of what makes a 5 star game different from a 4.5 one. Effectively, we are on a ten point scale, but I much prefer the stars than raw numbers. Why? Because when I was a kid, I obsessively read Leonard Maltin's Film Guides (it's true), and he used stars. Also because one of my game reviewer heroes, Tom Chick, uses stars. So that makes sense to me.

Now, I'd also like to elaborate a little on our rating policy while I have your eyes. As Editor in Chief, I very much encourage our reviewers to use the full range of stars, exercising prudence especially in regard to one star and five star games. I don't feel a five star range is valuable if almost everything is 3 stars and higher, nor do I feel that our mission here would have any credibility if we weren't identifying the real stinkers along with the really important, impressive titles that deserve the full marks. And if we dish out the five star reviews for every pretty cool game that comes out, then we are not really identifying the top-rated titles.

Some examples of five-star quality games.

When one of our writers puts across a review of one star game, then that is an article I am going to pay special attention to while preparing it for publication. I want to make sure that the lowest mark we give is supported by the content, and that I feel the reviewer has justified their position. Likewise, with five star reviews I need to see where the reviewer has made their case for the title earning it. A game being really fun, really cool or "my favorite deckbuilding game" is not enough. A five-star game needs to bring with it an impact, significance above the reviewer's favor; it needs to be a game that represents a moment in the industry, among designers, or in its genre. This is also why very few expansions deserve, in my editorial opinion, five stars.

If you are super mathy and like graphs and stuff, you might chart that our reviews do tend to skew high. There is a reason for this, and it has nothing to do with us promoting games for a retailer. It is because most of our assignments are volunteered. Our writers tend to write about things they want to cover and have an interest in. I do sometimes issue assignments, but even in doing so I do my best to match the game up with a reviewer that I feel will be best equipped to cover the material. I would not, for example, assign our man Drew a hardcore GMT wargame because that's not his bag at all. But Kyle, on the other hand, might be champing at the bit for the new reprint of Wacht am Rhein so there is a logical fit there. But you will still see disappointments, titles that didn't meet expectations, and surprises. We strive to be as fair and honest as possible in all regards. But I do not want a writer covering a game that have little to no interest in or enthusiasm for- at that point, the review would not be serving the best interests of our readership.

With all of the above in mind, please check out our new review key and see how it applies to some of your favorite titles. You might find that your appraisal varies greatly from our reviewers' takes. We want this. We may miss the mark, as far as popular opinion goes, or we may rail against it. Or we may join in the effusive praise for universally acclaimed titles like Gloomhaven. As professional reviewers, we want our work to challenge your views as much as we want you to nod in agreement with our work. We are not the arbiters of what is good and bad, we are critics. Every review is a dialogue in which we are engaging the reader and encouraging discovery. Hopefully, our star ratings serve as a guiding light on that path.