I think there is a fine distinction between 'Creepy' games and 'Scary' games. The former category uses psychology to disrupt the players sense of security and really tweaks their mind. This is my favourite form of horror and it really feels immersive and genuinely scary enough to prevent you from getting a comfortable sleep that night. Then you get the latter category which merely uses horribly mutated beasts jumping out at you from cupboards, all but yelling BOO, and getting blown away by the nearest twelve-gauge. This to me isn't horror, it's just making you jump. You know what else makes me jump? Someone running up behind me and poking me. But you don't see a game about that.
A classic example I can cite is the F.E.A.R series. The original F.E.A.R remains my favourite horror game so far because it does the 'creepy' thing so well. At any point the Antagonist, Alma, could pop up and scare the socks off you. The fact that she's just a little girl with no clear deformities makes it all the more creepy that she'll be standing in a pool of blood. One memorable event was when you're crawling through a vent and your torch will flicker. It'll come on briefly and you'll see Alma crawling towards you, then it'll go out again and it takes a little while to come back on again. Then... There's nothing. You're in the vent alone. That creeped me out.
F.E.A.R. gave me a different reaction. Once I found out that the "creepy" bits are never actually hazardous, I just kind of shrugged every time Alma tried to scare me. Sure, I might have flinched reflexively, but I never really felt the jolt of a good jump scare. In Perseus Mandate they finally forced you to contend with some of the things that jump out at you (or pull you unexpectedly to them) and as a result I found it creepier.
This is an area that I think interactive entertainment can surpass non-interactive entertainment in delivering scares, and it's part of this article: you, the player, are responsible for handling whatever the game throws at you. I say "can surpass" because it seems rare that games actually live up to this potential. Nevertheless, the potential is there.
Realizing it, I think, will require throwing out the idea of making an action-horror game. Despite the number of titles that fit that genre moniker, I don't think the two fit together particularly well. An action game usually wants to make you feel empowered, stoic, capable. A horror game usually wants you to feel vulnerable, edgy, frightened. If not altogether incongruous, these goals are at least inconsistent.
I also think it's important to distinguish "horror" from "terror." Although they're often used interchangeably, these words can mean quite different things: "horror" refering to the shock or revulsion felt upon seeing something disturbing, "terror" refering to the feeling of dread for an upcoming event. By these definitions, games are already very good at "horror," since that's mostly up to the audio and visuals. "Terror" though, often remains elusive.
In Dead Space there's a shambling pregnant woman with spikes for arms and a foetus bursting out to claw at you. It's a horrifying monster design. But it's not terrifying because I never dread it: if it jumps out, I'm always ready to dismember it with high-powered weapons. To be terrified by a game, I need to feel overwhelmed, vulnerable, and nearly powerless. Do any of those things sound like they'd make for a good action game?
What makes Fatal Frame more scary (to me) than Resident Evil or Silent Hill are two-fold:
1. Weapons. Sure, anyone can feel safe and secure with a rocket launcher or even a lead pipe, but what happens when you take away all conventional weapons and leave the player with something totally unconventional? If familiarity of the weapon is lost, in comes the horror.
2. The enemies. Thanks to the movie "Predator", we all know that if it bleeds, we can kill it. Zombies need special attention in shooting for the brain, but they fall like any other creature, and a couple whacks with the lead pipe can bring down a crawler in Silent Hill. But what happens when you are facing something that has no lore in mythology on how to kill? Ghosts are already fully dead and have no bodies, thus they cannot bleed or give us any indication on how to destroy them. Combined with the unconventional weapon given, Fatal Frame is definitely a fine choice for scares and thrills.