Company Predicts Game Review Scores With Science

Company Predicts Game Review Scores With Science

image

Everybody loves to play armchair game reviewer and guess what kind of scores a game might get before it comes out, but the folks at UK company Vertical Slice claim to have it down to a science.

Are we game reviewers really so predictable? Now I know there are some games out there that everybody assumes are going to get good reviews - I would be surprised if Halo 3: ODST and Modern Warfare 2 did poorly on the ol' Metacritic later this year - but come on, we can't be that easy to read. Right? British company Vertical Slice apparently thinks so very much: they're boasting that they can predict game scores up to a year in advance.

"People think you can't predict a game based on quantifiable data," director Graham McAllister said. "What we can do is get these estimators. Some people will just have a hard job believing it. We have analyzed the statistics to death, thorough and rigorous, and what we're saying is, 'You may not like it, but this is the best model that anyone has come up with to date.'"

The "estimators" Vertical Slice is talking about are some pretty heavy statistic and science work. Not only has the company "reverse-engineered" game reviews to see which phrases correspond to particular scores of reviews, but they've used intense biometrics to measure how people respond to games. After 30 seconds of play, they say, they can figure how where a game will fall in terms of its review score.

"Biometrics is our big thing; we hook people up to equipment that will measure your heart-rate or skin response," McAllister said. "If someone says, 'This is the scariest game ever,' we'll be able to say, "Really? Well, we don't think so." And we'll be able to prove it."

The point isn't to make game reviewers look like idiots, though. It's to help developers figure out sooner how they make their games, see where their games are going. "There's no reason why you would not want it," McAllister said. "The return on investment is potentially huge. At the minute, our clients range from PS3 developers to iPhone developers."

[Via Eurogamer]

Permalink

SO have they predicted anything yet? any previous predictions that turned out right?

ok...

"There's no reason why you would not want it"

Hmm... well there are plenty of developers who could care less about review scores. Also, having a formula to predict review scores might push publishers to force developers into a "stay-in-the-box" mentality, reducing innovation. In short, developers will be less likely to take chances at making something new.

My 2 cents.

HAH! Most gaming media can't predict if a game will be even generally good or bad the week before it comes out.

(Seriously GI? Seriously? You thought Too Human was going to be the RPG of the Year?)

Toty54:
SO have they predicted anything yet? any previous predictions that turned out right?

This. Thats what I want to know. They cant really make any claims if they haven't predicted anything yet.

How do they want to predict the games "up to a year in advance" when they need someone to play it? I know no game that was finished, much less polished nicely that far in advance. A year in advance most games aren't even playable.

Bollocks.

Half-Life. 5 minutes of doing nothing at the start.

Review? Maybe 75%.

RL: Game of the Year from almost all the magazines.

You know an aggregator site like Metacritic is fucking the game industry over when a company pulls the exact same stunt 2 times on 2 of its games.

Obviously I'm talking about Eidos and their review score fixing of Kane and Lynch and Tomb Raider Underworld.

I won't knock it until they prove it.

"There's no reason why you would not want it," McAllister said. "The return on investment is potentially huge. At the minute, our clients range from PS3 developers to iPhone developers."

Who are these developers? ...because I have this amazing snake oil to sell them. Yup, it can cure shitty games and make them into winners. Honestly.

Good grief. This is the most creative consulting scam I've ever heard of.

Well, in all fairness, some things ARE quite predictable. If a can of poop had "VALVE" stamped on it, it would be hailed as the next Game of the Year. If a dead carcass had "Blizzard" on the side, people would flock to it. The list goes on...

This sounds cute and all, but I'll wait for the real reviews before I decide to go play something. Good idea though.

The_root_of_all_evil:
Bollocks.

Half-Life. 5 minutes of doing nothing at the start.

Review? Maybe 75%.

RL: Game of the Year from almost all the magazines.

Yeah, but there's something about the atmosphere in that opening, a sense of interest and slight bewilderment. If that shows up on biometrics and they've done their groundwork well enough to factor in games like that, then it's likely closer to the accuracy they suggest.

Also, I separately suspect that in such a case, they would begin timing the "first 30 seconds" from the initial moment a fail-able challenge that pertains to the core of the game (ie. combat) occurs. Everything else is simple exploration and switch-flipping in a risk-free environment - certainly not the focus of the Half-Life games.

I call BS on this,if only for the reason that they're treating game reviewers like guinea pigs.

Also, they can't predict every game score. Infact, I bet they won't be able to get half of them correct. They're likely just people who think they can predict that Modern Warfare will get sold out or that Halo ODST will be nominated for GOTY.

It's old news. They're not special, it's just common sense.

I'm not gonna totally knock them. Developers have used biometrics for creating better games for a while (On Halo 3, Bungie used a biometrics firm to help figure out level design and enemy distribution). So, why wouldn't they be able to do this? But, my one concern is that if you predict something like this a year in advance, it might be a case of self-fulfilling prophecy. After all, if I told you a game was going to suck a year in advance, and that I had the science to prove it, when that game came out, you might remember it and also think that it sucks, even if it doesn't suck as much as I said it does.

Could be very useful to developers if this is actually accurate. May make for some interesting experiences in the future.

yeah i don't see how this will work, i think it's going to fall flat. tho i guess it depends on what 30 seconds of the game they play

The_root_of_all_evil:
Bollocks.

Half-Life. 5 minutes of doing nothing at the start.

Review? Maybe 75%.

RL: Game of the Year from almost all the magazines.

And yet in some sense, that first five minutes was the most representative of the game as a whole. "Okay, you're on a one-way linear track that you can't do anything about, interesting things are going on all around you, and though you won't have the time or ability to really see them or figure them out, if you just manage to hang on long enough, you'll reach the end just fine."

To be honest, I never got the big hullaballoo around half-life. Maybe because I played it only shortly before HL2 came out, or maybe it was because by the time I'd played it I'd been long steeped in things like System Shock 1 & 2, Thief, and the pure raucous fun that is Serious Sam, when I finally got around to playing half-life, I swear the most interesting bits were the initial 5 minute preview section and the cockroaches that ran from the light.

Being honest, I found the story as "meh" with the writers using the cheap but effective trick of leaving large chunks of it out so your own imagination could make it up, and the gameplay itself couldn't really get more linear. And then there's that final boss battle, about as hackneyed a boss battle as ever was. "Don't go after the boss directly, first you gotta take out these things that don't seem to have any relation, and THEN you can attack them in their vulnerable spot, using platforms and the like that seem to have no purpose other than to provide you the necessary vantage points"

But man, those first few minutes really gave you the impression that something cool was going to happen, didn't they?

Better yet they could use a time machine!

SCIENCE!

Wow, some people will believe anything. I wonder how much drugs you have to do to fall for something like this. :)

In theory I suppose it makes a degree of sense. As unpopular as they may be things like Sociology and Psychology work under similar principles and are perfectly sound and functional even if the results are often too politically contreversial in many cases to use to any real benefit (ie changes to long term behaviors and such on a massive scale generally aren't going to happen even if someone predicts 'if this continues, this will happen' correctly. No politician has the guts to run out with a punkhammer to force massive change either, risking becoming a tyrant in the process).

In practice however your dealing with a fundementally corrupt industry where there is generally only one variable: Money.

This was proven by the whole Kane and Lynch fiasco. Basically a game that buys a lot of advertising is NOT going to be reviewed poorly by the critics working for those advertising sources. You also have things like the recent "Arkham Asylum" fiasco, where despite it being a big game, it was made pretty obvious that by giving it a high review publications could jump the date for reviews and talk about the game early and probably sell more issues as a result.

Then there is the entire Grand Theft Auto IV thing, that game got a 10 despite the fact that VERY few people felt it was worthy of a "perfect" score especially given numerous, obvious flaws that cut down on the enjoyment of the game for pretty much everyone. The whole thing with your fat cousin Roman calling you every 15 minutes (along with your other buddies) has become something of a universal gaming joke. With something that annoying and poorly implemented in the game there is no way it should have ever become "perfect".

On the other hand Microsoft paid Rockstar big bucks for exclusive content, and some of those bucks doubtlessly went towards "cooking the books" with reviewers.

Generally speaking your not going to really be able to predict scores based on any kind of analysis of game trends, and comparison. Money and industry corruption is going to always be a factor and there is no way to universally tell who is doing what.

I could make a total piece of liscence cr@p but if I toss a million dollars to Game Informer, I'd be bloody surprised if they didn't give me a '10' claiming that my game is the most polished and revolutionary thing ever.

Kwil:

To be honest, I never got the big hullaballoo around half-life.

Half Life was the first time a FPS really had a storyline apart from the action.

Admittedly, if they measured the biometrics during the resonance cascade, they might get a huge surge, but trying to measure Peggle, Team Fortress 2, Portal and GTA against any sliderule seems doomed to failure because they're such different games; yet they all brought home massive ratings.

Are we game reviewers really so predictable?

Yes.

You don't need biometrics to predict Metascores. Reviews write themselves these days: "Satisfying combat... clunky controls... convoluted plot".

Everyone knows that Modern Warfare 2 will get 90+ reviews, regardless of its quality.

Not to mention that certain genres are excluded from higher scores for no apparent reason.

They predict review scores...what, do they have an inside line on publishers' marketing budgets?

All I can see coming out of this is

"You're game is bad"

"Lets push it back another 6 months!"

Statistics-wise, how much weight is given to the amount of advertising funding the game was given? Particularly to review sites and magazines? Because I'd wager that that has as much to do with Mediacritic scores than anything else.

Edit: Also, a lot of you seem to be missing the point. This program wont tell you whether the game is good or not, or if it will be a commercial success. It's supposed to tell you what it's Mediacritic rating will most likely be. And we all know good review scores != good game, or bad review scores = bad game.

Having a working knowledge of statistics, I don't have any trouble believing you can predict those scores. Something that is more mainstream, has more brand recognition, and has a fanbase that will burn down the headquarters of a publication that reviews games if said publication gives it anything less than a 9 will always tend towards higher review scores.

Toty54:
SO have they predicted anything yet? any previous predictions that turned out right?

Yeah, I'd like to know this little fun fact, too.

I don't know. This smells like pseudoscience to me. One of the 14 indicators is speaking directly to the public, and they're trying to sell something.

Something smells rotten.

pimppeter2:
All I can see coming out of this is

"You're game is bad"

"Lets push it back another 6 months!"

Don't give them ideas @_@

"Yeah man we hook your brain up to Biometrics and the screen flashes colors and then we tell you how much your game will make on metacritic. It makes perfect sense right?"

If this works, I will kill myself because humanity is so damn predictable.

Of course it'll work. It should be relatively simple.
Big game releases with a lot of hype Assassin's Creed II, MW II,ect. will get good scores.

I'm just saying... You can watch a trailer of a game a lot of times and be able to say, "Wow, this is gonna blow"

olee12343:
ok...

"There's no reason why you would not want it"

Hmm... well there are plenty of developers who could care less about review scores.

Couldn't care less you mean.

VanityGirl:
Of course it'll work. It should be relatively simple.
Big game releases with a lot of hype Assassin's Creed II, MW II,ect. will get good scores.

I'm just saying... You can watch a trailer of a game a lot of times and be able to say, "Wow, this is gonna blow"

True, but you can also think "this is going to be great" then realise that they chose the five best bits of the game to show in the trailer and they conveniently missed out all the terrible parts.

VanityGirl:
Of course it'll work. It should be relatively simple.
Big game releases with a lot of hype Assassin's Creed II, MW II,ect. will get good scores.

I'm just saying... You can watch a trailer of a game a lot of times and be able to say, "Wow, this is gonna blow"

Im not even pretending to be a genius when I say I could do what they plan to do on my own with only youtube and a few magazine subscriptions.

If some publishers were willing to take my word for it then I'd be minted by now.

 

Reply to Thread

Log in or Register to Comment
Have an account? Login below:
With Facebook:Login With Facebook
or
Username:  
Password:  
  
Not registered? To sign up for an account with The Escapist:
Register With Facebook
Register With Facebook
or
Register for a free account here