After a man ejaculates the sperm it usually takes between 24 to 36 hours for the sperm in the man to refill.
However when I've masturbated before I could masturbate and ejaculate again within a few hours so I guess some men can refill their sperm within a few hours.
However the sperm that comes back may not be good enough to create a baby and get the girl pregnant.
But it's possible to have sperm come back to a man within a few hours from my personal experience but doctors say that it can take some men between 24 to 36 hours to get more sperm inside them.
But that may be for the sperm to come back that is good enough to get a woman pregnant.