The problem was that my mother was especially impressionable by people at her church, and, once she found me poring over books that depicted evil statues on the cover and line drawings of monsters and bare-breasted demons, she quickly confiscated them lest I start my own cult. My friends at the time weren't interested in pretending to be fantasy heroes, so there was little point in sneaking around to play D&D.
Years later, I moved to New York City and got married. I was still into fantasy and read a lot of novels with dragons or swords on the cover, which my friends affectionately called "Dork Books." Surrounded by millions of people and living hundreds of miles from my mother, I realized that there was nothing keeping me from playing the game that fascinated me as a kid. I grabbed the 3.5 Player's Handbook and a bag of dice and started playing D&D every week in the Upper West Side.
When I mentioned my plans to play Dungeons & Dragons to anyone else, including my wife, I was confronted with the same reaction: "What the fuck are you doing that for?" I was an "adult" and playing D&D was for "socially retarded losers." My wife was especially incensed that I chose to spend a night away from home in order to pretend to be an elf in front of strangers. But she was an actress, was her chosen profession that much different than my hobby? My circle of friends and acquaintances was generally liberal (this was NYC after all) and many of them played videogames, even RPGs, but, somehow, the idea of sitting around a table and playing something similar was abhorrent to them.
I think that the media backlash is the easiest culprit. In 1980, D&D was a fringe hobby not unlike model trains or stamp-collecting. These hobbies, however, were never associated with a crazy guy who lived in the steam tunnels under the University of Michigan on the 10 o'clock news and on shows like 20/20, which shaped much of our public opinions back then. The scandal occurred at a time where the RPG industry was the most vulnerable. There was no internet to give voice to the tens of thousands of gamers who were demonstrably not crazy. TSR did nothing to combat the growing smear campaign, hoping that, by ignoring it, the hysteria would simply go away. As a result, in a very short time, the only feeling that most non-gamers felt about D&D was negative. "Dungeons & Dragons? That's bad."
In the 90s, the hobby was pushed further into the fringe. The only people who risked being known as tabletop gamers were likely social pariahs already. Unlike the 80s, there was little crossover from the somewhat more popular gifted kids to the socially stunted kids who played D&D. I knew them in high school. I would play Magic with them sometimes during lunch, but, unfortunately, they reinforced some of the gamer stereotypes that still exist today. They didn't shower often; they didn't go to the prom, let alone kiss a girl.