I made a post about this a while ago. Short version, you need a competitive/ranked queue to be fundamentally different than the casual/normal queue:
Draft picks and bans, no legendary items (possibly no items at all), known rankings instead of just a hidden ELO, etc.
All of that and more. Players need a very real and tangible reason to pick one over the other, instead of just “whichever has the shortest queue times.”
I do like your idea of limiting large groups to the competitive queue, but that alone isn’t enough, IMO.