Microsoft is aware that the Xbox One party system doesn't work like it should.
Microsoft is still working out the kinks of the Xbox One operating system. Of the problems discovered by early adopters, which range from problems connecting controllers to malfunctioning disc drives, some of the most common have revolved around Xbox Live party chat on the new console. Voice of Xbox Larry Hryb, AKA Major Nelson, addressed the issue yesterday, responding to Reddit thread about Xbox One party chat problems:
"I had a meeting today about much of this and I can say that things will get better," Hryb wrote. "I can't offer a timeline of a list of what till be addressed first, but we are aware of the issue and things will get better."
Reddit user Numb3rs4 describes the current functionality party system as "backwards." Some Xbox One players report that they can connect to Xbox Live, but can't connect with others in a party. The issue can go so far as to prevent those players from hosting or even joining multiplayer games.
If you are having problems with your Xbox One, you should report them (and look for a fix) on the Xbox One support forums.