Hey, what do you do when you are out of ammo? Do you fly home or continue with rest of squad? And what if you are the leader, does the rest of squad follow you home or they continue alone?

You can issue the "Split formation" command "S" which will make one of them take on leader and continue.
If you are out of ammo go home or land at nearest one of your airfields. It's the safe thing to do

If you're out of ammo, you'd better fly asap to the nearest friendly airfield... If you are the leader you can also call for a "washout" with Ctrl-R. Every plane will drop the mission and fly back home on its own.

If I am over friendly territory and the mission is "patrol enemy lines" I abort and have everyone go home. The reason is if the flight continues without me and gets into a scrap over enemy territory and I land and end the game, bad things can happen to my flight. If I am over enemy territory and run out of ammo, I get out the best way I can. My mates will not be far behind by the time I get home. Hope that makes sense. The only time I may split off and let my flight continue is "patrol friendly territory."