Then there's the issue of hat selection. Too often men make the mistake of choosing a hat they like, rather than one that looks good on them. Not every man can wear a trilby in the same way not every man looks good in vertical stripes or boxy glasses. A hat should compliment your head and face shape, and the color should do the same for your skin tone. Black, for example, might look too severe on a light-skinned person - particularly paired with a black coat.
And yes, if you're wearing a fedora - especially a black one paired with a trechcoat or a dressed-down look - people may assume negative things about you. Women have largely pegged this uniform as belonging to the so-called "nice guys" that think being decent to a woman is a ticket into her bed. I'm sorry about that, but it's true. (For what it's worth, I asked two women outside the gaming community what kind of men they associate black fedoras with. Answer: Creep, avoid immediately.) Clothing is a representation of ourselves, and if you're wearing clothes that are associated with a particular subculture or stereotype people will, fairly or not, form an impression based on those clothes. It's true that wearing a fedora doesn't make you a Reddit-dwelling Men's Rights Activist in the same way that popping your collar doesn't make you a bro, but in both cases you're taking a risk by adopting the stereotype along with the style. About 50% of fashion's social function is to influence how others see you, and the impression a fedora creates often isn't positive - if you didn't know about this stigma, you do now.
However, fashion's other half - I'd argue the most important half - is about influencing how you see yourself. And you know what? If wearing a fedora makes you feel good about yourself, do it. I wear one at my desk on occasion, an Indiana Jones model my wife got me for my birthday. It doesn't suit my face and I'd never wear it in public, but it reminds me to live the adventurous life I've always wanted. (Though I can tell you from personal experience fedoras perform poorly on an archaeological dig.) Sure the fedora's an extreme fashion statement, but in a culture that considers cosplay to be appropriate attire for a con-goer and even celebrates it as body-positive, I don't see why, if done properly, fedora wearing should be a cardinal sin. Both attempt to craft the self into an ideal form.
So to those that would judge the fedora, I say: I understand. There's certainly overlap between the self-proclaimed "nice guys" of the internet and the fedora fanbase. But ridicule is a poor motivator for personal change, and it would be more productive to target the problem behaviors rather than fashion choices.
To those that cleave to their hatbands and snap-brims, I urge you to do what makes you happy - I'm not here to police your body. But also know that wearing a hat is a bold fashion move that carries inherent risk that you'll look behind the times or even come off as a stereotype. If you do want to wear a hat, buy it at a decent shop, since they'll help size it properly and find a color and style that compliments your appearance. Hats are fun, and there are hundreds of styles to choose from, so don't feel boxed into one look. Try a pork pie or a Panama, experiment with colors and materials. After all, if you're going to make a fashion statement that bold you might as well wear something different than, you know, everyone else.
For my part, I'll stick to scarves, good shoes and the occasional watch.