Having a fear of small children, infants etc is called pedophobia.
The cause of pedophobia is not really known but it's been suggested that pedophobia is most likely caused by the embarrassment that children especially small children can cause to people.
Some people like a friend of mine says he hates being around small children because it embarrasses him when a child does something cute or funny that he used to do as a child.
It's especially true when his parents are with him and his parents then embarrass him further by pointing out that he used to be that small and used to do those things such as eating dirt, playing in the sandbox or running around outside in his diaper.
He gets really embarrassed when he's with his parents and there's a small child running around in just diapers because he used to do that as a kid himself so embarrassment could be a reason that some people want to avoid kids and are fearful of them.
He said that he never wants to have any kids because they cause him too much embarrassment so that could be the reason for most people having a fear of kids.