DARPA hopes to make crowdsourcing an exercise in entertainment.
We at The Escapist love the Defense Advanced Research Projects Agency (DARPA) . Since its inception in 1958, DARPA has been actively developing truly innovative technology and pursuing cutting-edge scientific research. We’ve naturally covered a lot of DARPA’s goings on over the years because of its fondness for robots and similarly amazing technology. Just last year it funded Boston Dynamics’ development of the brick-throwing robodog and “WildCat,” which makes the Four Horsemen seem quaint, as well as making strides towards the inevitable future of combat, jet-mounted lasers. (PEW PEW!) DARPA is basically XCOM‘s R&D team.
Most recently, DARPA turned its Sauron-like gaze on gamers, sponsoring the development of several browser-based titles intended to test military software vulnerabilities. As part of its “Crowd Sourced Formal Verification” program, DARPA worked with a handful of game developers to create entertaining ways to solve real life software testing problems. I had an opportunity to discuss the project with Aaron Cammarata, the Chief Creative Officer at one such developer, voidAlpha, which was responsible for StormBound. If you’d like the whole story, check out the full interview here.
The obvious question, of course, is how in the world you end up developing games for DARPA, which is best known for things like the aforementioned weaponized robopuppy. (Okay, maybe “weaponized” is a strong word, but those concrete blocks could do some serious damage!) It turns out that Cammarata had actually worked with DARPA before on an unrelated video game project. After hearing about the project, “… honestly my first response was to go look up Formal Verification. I have a degree in CS, so I had a vague idea in mind, but trying to fit that together with ‘video game’ was making my head explode,” Cammarata said. Bringing a single-slide presentation to a “Crowd-Sourced Formal Verification Proposer’s Day” in San Francisco, Cammarata pitched working with the studio to “a room full of really scary-smart people.”
I was also curious as to where the notion behind StormBound came from. Crowdsourcing has been around, sure, but it’s traditionally been thousands of people poring over satellite images looking for something, rather than each person playing their own personal game (Unless you consider “I Spy” a form of high entertainment, I suppose). DARPA, it turns out, decided to innovate crowdsourcing on its own. “The project’s parameters were set by DARPA’s goals, and we did our best to provide a game that would meet those needs,” said Cammarata, going on to ask “If these kinds of problems can be crowdsourced, what else might we solve this way?” He even went so far as to suggest that this is “the birth of a whole new industry,” calling StormBound “a game that serves a functional purpose.” It goes beyond the now-outmoded idea of “gamification,” which typically aimed to make tedious tasks slightly less tedious – or even addictive and attractive – by introducing reward mechanics. StormBound and its ilk actually present a fun and challenging game experience, and applies the user input to solving legitimate Computer Science problems.
Sure, games have been considered for use in military training, successfully used in rehabilitation efforts, and even used to supplement traditional education, but, at least to my knowledge, nobody’s ever mapped solutions of dynamic puzzles in games to real-world software testing. Will it be a billion dollar industry? That’s all going to depend on adoption, I expect. If DARPA can make the next Candy Crush, it will invariably be testing a lot of software.
StormBound starts with a tutorial which explains the basic concepts for solving the puzzles. As soon as you’re out of the tutorial, however, you’ll start working on real puzzles, offered up by the testing servers. Cammarata would love to be able to offer “live data” even earlier on in the process – in the tutorial itself, perhaps – but “The problem is that it’s a catch-22,” he explains, “this isn’t like a Sudoku or crossword, because we don’t know the answer to a level before we give it to you. That’s why we need you to play it!” I was also curious about the division of development time between game and verification. I was impressed to learn that it took only 18 months to go from concept to execution. “A lot of that time was building the bridge between gaming and verification.” That almost goes without saying, given the fact that verification requires a highly-specialized skillset, and voidAlpha was essentially playing Gutenberg here, allowing literally anybody to participate in the process. “That’s what makes it a capital-R, DARPA-hard Research problem,” Cammarata said.
If you’re curious about how the game-to-verification integration works, you should check out Jef Bell’s explanation. I won’t pretend to understand it entirely, but the basic idea is that the game will serve you a bit of software data, presented as a puzzle. You “analyze” that data using the tools provided in the game. “The spells you cast in StormBound describe the patterns,” Bell explains. By combining the spells (in Formal Verification they’re called assertions) used to clear a level, it creates a way to prove the truth of Boolean (true or false) properties of the software.
While I can’t necessarily explain it all in plain English, nor can I really understand the nitty-gritty details all that well, one thing is clear; this concept has the potential to change the world of formal verification for software makers. In its present form, it’s not going to be the next Plants vs. Zombies or Peggle, but as the burgeoning CSFV industry moves forward, refining the ideas and creating even more compelling stories and mechanics behind the verification-driven games, it may be a simple matter for commercial enterprises to start utilizing the practice to better consumer products as well. Can you imagine a world where Windows isn’t riddled with vulnerabilities? Neither can I, but given the track record, I suspect this is our only hope.
How did you end up working with DARPA for StormBound?
My background is in entertainment games, mostly in the design field, on some AAA titles and big licenses. At our last studio, my partners and I had worked with DARPA on another video game project, and we knew that working with them offered the chance to tackle really hard, satisfying problems. I am sure our experience in ‘traditional’ video games helped get us onto the project.
We heard about a CSFV (Crowd-Sourced Formal Verification, the DARPA program name) Proposer’s Day that took place in the SF Bay Area, where we are located. When my partner, Mark Day, described what the project was all about, honestly my first response was to go look up Formal Verification. I have a degree in CS, so I had a vague idea in mind, but trying to fit that together with ‘video game’ was making my head explode. I happened to be reading a book at the time with a similar concept, and I thought “I have GOT to check this out.”
I quickly put together a one-slide presentation and got up in front of a room full of really scary-smart people to pitch them on working with our game studio. My goal was just to get through the day without saying something completely stupid.
After that meeting, we had several people reach out to us to explore partnering. We ended up working with Portland, OR-based Galois, Inc., which has been an absolute dream. They are leaders in the verification area, just incredibly bright folks. As a team, we really see eye-to-eye on the product, and there’s been a great synergy between the companies, which I think has resulted in a project that is more than the sum of its parts.
Was the idea to crowdsource complex problems with a browser-based puzzle game yours, or did you get the task of making StormBound?
The idea to crowdsource the problem was all DARPA, and our answer to that challenge was StormBound. The project’s parameters were set by DARPA’s goals, and we did our best to provide a game that would meet those needs. It is been an incredible design challenge, to come up with a game that accomplishes ‘real work’, and we’re interested to see where it goes.
If these kinds of problems can be crowdsourced, the obvious next question is, what else might we solve this way? Could neural networks be improved more quickly by re-using the ‘train your Gola’ idea? The concept was chosen very carefully and intentionally, and it could be extensible.
Frankly, I think we are seeing the birth of a whole new industry, or a huge new addition to the game industry. First came gaming as entertainment, then there was edutainment, then serious games that try to train or raise awareness or improve social consciousness and then gamification – the sugar-coating of boring tasks to make them more palatable. StormBound is on the leading edge of that vector. It is a completely new type of product – a game that serves a functional purpose. It is going to need a new name; I’ve been using “utilitainment.”
Are the levels dynamic, or are the puzzles always the same on any given level?
Actually, yes, the levels are dynamic, and that’s one of the most fascinating pieces of the system to me.
When you solve a level, it gets passed up to the server, which does incredibly hard, PhD-level math on it, and gives you back a result. Either your solution to the level was just what we needed to do some verification work, or it wasn’t. If it wasn’t, the server changes the level, and challenges the player back with “Ah ha – but what about this?” The player is notified that their solution wasn’t quite right – in the game, that’s called a Fuse Breach. The players then get to go back into the level and refine their answer.
In a way, your relationship with a level is almost more like in an RTS. You don’t just solve it and move on, like in most puzzle games. Instead, your solution to the level is ‘active’ until the server gives you the thumbs up or down. For a while we actually toyed with a scoring system that took this into account – the more answers you had, and the longer they stayed ‘alive’ or valid, the more points you’d get. It was almost like mining in an RTS. In the end, though, we couldn’t get the balancing and messaging right in the time we had, but it’s something we might explore again.
When you look at it from the 10,000 foot view, that’s a really mind-blowing thing – this is the first puzzle game with a “Game Master.” The server is in a dialog with the player to literally change the game based on the player’s actions. That is really an incredible concept, especially when you consider that the result of that dialog is to perform formal verification, to serve a real-world need that will benefit people. That just blows my mind!
Is there a certain point in game you have to get to before you’re actually contributing to the testing process, or do low level puzzles offer some benefit as well?
Right now, yes, you need to finish the World Map tutorial before we give you any live data. One of our goals is to lower that bar so you can contribute from very early on in the process.
The problem is that it’s a catch-22 – this isn’t like a Sudoku or crossword, because we don’t know the answer to a level before we give it to you. That’s why we need you to play it! So what we do instead is look at some of the properties of the level, and give you only levels you have at least a chance of solving.
For example, if you haven’t done the Etheric Basics tutorial, you must skip levels that have Akashic Streams. It would be impossible for you to solve them, because you can’t even get into Etheric mode. Beyond that, though, we need you to at least be proficient with the three Basic Akashic patterns (Stack, Above, Tangle), and Compound patterns. Once you can do those three, and can combine them, you can solve any level with at least one Akashic Stream, so we start giving you real levels right away.
How long did development take, and what portion of that was dedicated to figuring out how to actually use player input to solve real problems?
We have been working on the game for about 18 months now, from initial concept. A lot of that time was building the bridge between gaming and verification. There were a lot of in-person meetings for us game folks to understand exactly what the verification team needed back from the game, and to help the verification team understand some basics of game design.
There were definitely a few false starts, which is par for the course on a project like this – that’s what makes it a capital-R, DARPA-hard Research problem.
We finished building the first version of that bridge about three months into the project. We actually called it our Golden Spike milestone, like when they connected the first Transcontinental Railroad. It was the first time a player could load a level that came straight out of the verification server, solve it through gameplay and send the answer back. It is one of the real successes of the project that we got something up and running very quickly, as that gave us a long time to iterate. We have probably built and thrown away three or four game concepts along the way, that’s how challenging it’s been to find something that will work – and be at least somewhat entertaining.
In whatever detail you can provide, how exactly does solving a puzzle in StormBound assist in testing software?
If you want a fantastic, detailed description, I recommend Jef Bell (Galois)’s excellent blog post, here.
What motivated the story part of StormBound? Has the player reaction to the story or the gameplay been more pronounced?
We were trying to attract a particular kind of audience that enjoys hard puzzles, and made a decision early on to create something that would hold players’ attention. We wanted to give people something cool to sink their teeth into, that they’d want to stick around and watch unfold. We are really trying to build a long-term relationship with the audience.
To create the story and the world, I reached out to a long-time friend, Christian Gossett, comic book artist, author and award-winning director. He has created entire universes before, so I knew he would help open the window into this new place we were creating. We are just getting started with the story – we actually have a long-term story arc document that we are hoping to use over the coming months and years to give players a much richer view into the StormBound world.
Player reaction has been an interesting mix – some players love that there are these cool new concepts to explore. Others have found that it makes learning the game a little harder for them, that some of the names for things were a bit too much. So we’re going to work on making it a little easier to swallow, especially at the start, because we think the magepunk theme has a lot to offer, and will give players a real sense of place and community within a very cool world.
What kind of player base do you need to achieve in order to really make this project work?
It is still too early to tell – this is just our very first release, and we will be making improvements as we iterate on things. We are still learning the best way for players to play in terms of accomplishing project goals. I will say that so far we’re actually way ahead of where we thought we would be, and it does not need to be a top-10 social game to be effective.
Where are you now with your player numbers, if you can share that information?
We can’t share specifics, but I will say we got a fantastic turnout at launch, which I am very happy with, and we are getting way more interest than we had forecast. I would say we’re on par with a very early-stage beta online game, which is really what we are. Right now we’re in a mode of engaging with our players to hear what they do and don’t like, and making the game better for them. We will be working on growing the audience over the coming months, by offering a better game than we have now, with the added benefit of helping solve a real problem.
