Yep, known unresolved issue on XB1. Gory details here:
TL;DR - join a random game then immediately quit and join your friend’s game. It may help if they are waiting at the game menu rather than actually in-game at the time.
I strongly suggest any follow-up go in the linked thread, just to keep it all in one place (@Derch, @Ganjamira, someone?)