I'm going to be totally candid here; if your new job isn't allowing you much time, then to be honest, I'd either find someone to loan him or sell him. He sounds like he needs lots of time, patience and groundwork, but if you're unable to give him that, then no quick fix will solve it, I'm afraid.
If you are able to put time aside, then my first thing would to be earn his trust. You say he is scared of men, well presumably he's had a bit of a bad past and perhaps experienced heavy hand riding, hence the constant desire to avoid contact but also the panic he goes through.
If it was me, I'd not be riding him, sometimes with horses we have to remember that they aren't born knowing everything and when they are messed up, we have to be able to spend time fixing those issues in the most empathetic way possible.
I'd start by just earning his trust, grooming him, spending time in the field/stable and just enjoying him. Then I'd begin by doing lots of ground work- long lining him and in-hand work. The in-hand work will really help you see him on the ground and is great for bringing out a horses confidence. You aren't at risk of being thrown off, and you can focus on establishing a contact from the ground. You will have to be really light with him, no punishment etc.
As he responds to voice, you can use this to your advantage. Praising him for being responding well to things, I'd recommend getting an in-hand training book, Oliver Hilberger's is fantastic and also seeking veterinary/dentistry advice, to perhaps try and get to the root of the problem, my guess is that he's had some serious mistreatment in the past and been exposed to firm/rough handling and is a very sensitive horse.
Be patient, take your time and don't worry about riding him yet, just focus on his issues and take it slowly.