So this, then, is the paradox of gamemastering: In order to make sure that everybody could have fun, you have to be willing to let the players make choices that lead to results that aren't fun. You can't guarantee the fun. And if you try to make sure everyone has fun, eventually you'll guarantee that no one has fun at all, because you'll destroy the sense of agency which is the root of the hobby's pleasure.
Agency and Causality, Or Why Rules Matter
The agency theory of fun also explains why rules matter. Rules, in a tabletop RPG, are ultimately about what philosophers call action, where "action" means intentional effects caused by an agent. It is the rules that dictate the results of action, and thus define the relationship between a player's choices and the consequence he experiences. The rules provide the framework of cause and effect that gives meaning to choice. For instance, virtually every RPG has rules that dictate when you may choose to attack a target, how the success or failure of this attack are determined, and the consequences of each.
A game without rules cannot provide a sense of agency, anymore than a world without causality can. If the players operate subject to arbitrary outcomes - what the ancients called "Acts of God" and RPG designers call "GM Fiat" - they have no meaningful way of knowing or understanding what the consequences of their choices will be, and thus no agency.
I believe that the agency theory of fun is the reason that esoteric games like Amber Diceless Roleplaying or Everway have never caught on, and why as games evolve, they evolve in the direction of more rules. Comprehensible, detailed rules add to the player's sense of agency, just as playing with friends in an ongoing campaign does. (This does not mean that extremely complex games like Rolemaster are an unmitigated good - but that's a critique for another time. Let's just say that the simplest rule system that provides agency is best.)
Agency also explains why dice are, and will always be, a popular mechanic with RPGs. As I explained above with the example of the hidden artifact, if the consequences are pre-determined, then the choice is not real. The inherent contradiction between omniscience and free will has plagued religion for thousands of years, and it plagues RPGs, too. For instance, imagine if tabletop RPG combat went like this:
Player: "I attack the dragon."
GM: "Based on your attack bonus and the dragon's armor class, if you attack, you are certain to miss."
Player: "Uh... well I don't attack, then."
It's hard to imagine that game being much fun because the result of the player's choices is determined before he's made them. (This is the same reason that Tic-Tac-Toe isn't fun.) Agency, then, requires that we be able to predict the consequences of our choices, but not with certainty. D&D creates agency with its Core Mechanic: "To determine if your character success at a task, you roll a d20, add any relevant modifiers and compare the result to a target number. If the result equals or exceeds the target number, your character succeeds. If the result is lower, you fail." The relevant modifiers and the target number provide causality. The d20 provides uncertainty. Both are essential.
Don't Change the Dice, Change the Rules
Because randomness is inherent to RPGs, every gamemaster soon becomes familiar with the temptation to cheat, or in gamer parlance, "fudge the dice." For instance, imagine that a new player, Carrie, is joining your campaign. In her very first battle, her character takes a critical hit, and is killed. The temptation will be very strong to pretend that the die roll was different - that a critical hit was just a normal hit, or even a miss. Especially if you think "my job is to make sure Carrie has fun," you'll convince yourself that dying is not fun, and that therefore Carrie's character shouldn't die.