The mystery isn't whether or not the Legend of Zelda games fit together, it's how they fit together.
Nintendo games and timelines have an odd relationship: Some series, like Metroid, have very clear timelines, while others, like Mario, don't really need one. But then you have the Legend of Zelda series, which does have a timeline, just not one that anyone can agree on. In Issue 302 of The Escapist, Logan Westbrook looks at some of the difficulties involved with putting together a Zelda timeline, and offers a theory of his own.
For example, some fans believed that the green-clad hero in the various Zelda games was the same person each time, not unlike Mario in the Super Mario games. Unfortunately for the idea's supporters, Shigeru Miyamoto demolished it in a 2003 interview with Nintendo Power, when he said that the series actually featured a multitude of green-clad heroes called Link. Nintendo later confirmed this in the opening to Wind Waker, which refers to the legendary "Hero of Time," the title given to the Link from Ocarina of Time.
Unfortunately, the information that Nintendo allows to trickle out can sometimes raise more questions than it answers. In 1998, Shigeru Miyamoto - again talking to Nintendo Power - said that A Link to the Past came after Adventure of Link chronologically. This caused confusion amongst fans, because Ganon died in the original Legend of Zelda, and remained dead in Adventure of Link. No one knew how Ganon was supposed to have come back to life, and the back story for A Link to the Past made no mention of it. Even more confusingly, several pieces of promotional material for A Link to the Past said that it was set "generations" before its predecessors.
At times, it seems like Nintendo is toying with fans, making the Zelda timeline more convoluted and difficult to work out. You can read about it in more detail in Westbrook's article, "Ocarina of Timelines."