Randoms changing the map

The other day I was in the middle of lookin for some stuff and another player jus switched the map, right after joining.

Is there a setting to prevent this?

No but a popup (with chime sounds as it counts down) does appear in the bottom right of the screen. If you see that just bring up a menu to cancel fast travel. Also if you are the host, you can always kick the person who bothers you.