Take Lateral Thrusters at level 5 as it dramatically improves your base movement speed. In fact, it improves it so much that whatever sensitivity you normally use is going to be found lacking; I normally bump it up two or three notches whenever I play Caldarius in order to keep up with his movement.
Flashbang is excellent at forcing enemies to disengage but also probably the best kill secure in the game. If you come across an enemy who looks already likely to run, save it and either hit them as they turn to run (forcing them to usually start going in the wrong direction) or finish them with it as they escape.
Gravitic Burst has a push effect, but it can be canceled by holding L2 as you hit - the animation snaps into his sword swing, but it’ll still deal normal damage and you’ll tack on 150ish damage from the sword swing as well.
His lore gear is fantastic and encourages you to waveclear with his ult, since it gives you a tremendous chunk of lifesteal. This fixes a large weakness of his kit (no sustain) and lets you solo harass much more effectively. On a related note, his ult tends to clip geometry a lot in Overgrowth thanks to low-hanging roots; be very careful using it around big shard and at the choke, where he often accidentally hits a badly-formed corner up in the air.
Use Applause. It’s by far the most insulting taunt in the game.