People remember the play sessions where confluence of supply (material constraints) and demand (formal constraints) forced them into a state of sweet, sweet flow, where they hauled their ass in gear and pulled through, where their course of action seemed like it couldn't have been any other way. People remember their experiences of challenge as ... stories. The best examples of this tend to occur when the player feels some sort of social alignment with the parties involved, whether they're other players or well-constructed NPCs. I didn't just solve some esoteric puzzle, I helped Manny Calavera find final peace. I didn't just micromanage the hell out of a couple dozen military buildings and five score units, I helped them destroy the Zerg Overmind once and for all. I didn't just spend 100 precious hours of my life repeatedly clicking to build higher and higher stats, I teamed up with my fellow Horde and participated in a glorious raid. I haven't just spent my entire life as a lump of grey meat churning more complex electrical patterns in an endless requiem of learning and adaptation, I interacted with people.
We need to stop thinking of challenges as obstacles to be mastered, and start thinking of challenges as realities to negotiate. Social dynamics are the toys to charm society. Social challenge is what we'll call the feeling when we push through the climax of an interactive storyworld and look back on the very personal effect we had on our respective stories. The excluding factor in other forms of challenge is they force people to adapt to the system on its own terms, something many can't do, even if they had interest. But social maneuvering and choosing between socially created values and bonds are what people have been adapting to their entire lives. Release a socially challenging game and you've got a potential audience of hundreds of millions of people. You can't say the same thing for an RTS.
Social challenge seems difficult to imagine, much less implement, due to the finely granulated and fuzzy nature of social interaction. The raw tools are available: Context-specific pattern recognition AI, personality models, probability theory - there are many technical approaches to social challenge. Designing social challenges will typically involve importing or modeling a social mechanism from real life, whether it's a particular complex relationship, a family feud, a political revolution, the alienating halls of modern middle-schools - translating a culture to rules is essential to support the core paidia. Material constraints will consist of how characters express their personalities; formal constraints will consist of their motivations.
Since most people are able to handle social challenge well before they ever sit down to play, the pacing architecture can change from building up to higher and higher levels of difficulty, and toward building up major thematic choices. The public wants these sorts of choices presented to them, even if they don't know it. There is a buzzing transparent need beneath the surface of our culture, a desire to play with vital issues film and literature cannot bend to approach. Society needs to be challenged if the culture at large, perhaps humanity, is going to adapt and prosper in this insane world of accelerating change. If game designers take advantage of this brave new territory, in 20 years, pundits might just look at videogames as the cultural force that kept us all sane. Gamers aren't going to convert the mainstream to gaming - the needs of the wider market will convert gaming to the mainstream.
Patrick Dugan is a ludosophist. He runs King Lud IC, a blog regarding game design theory, memetics and interactive storytelling. He looks foward to prototyping with Chris Crawford's Storytron, and to pioneering socially-oriented narrative challenge.