Can't connect to friend online

So my friend and I are trying to connect to play co-op on the PS4, but we can’t connect when it’s just the two of us playing. Currently we can connect to the same quick match in match making and play just fine. The problem is we would really like to play with just the two of us.

If it’s just the two of us it will just tell us that either of us can’t connect to the host, the connection will timeout, or my favorite one reasons is that I need to check my network cables. For what it’s worth both of our NAT are set to Type 2, which to my understanding is perfectly fine to play together online.

Has anyone ever run into this? Does Gearbox know that this is happening? Is there a workaround or is something that they are planning on fixing? Are we SOL?

Any help would be greatly appreciated.