If you change a babies or toddlers diaper and wipe up poop without using gloves then you don't wash your hands really good after changing the diaper then you could pass the feces into your system through food when you eat.
But other than that you shouldn't get sick as long as you wash your hands with warm water using antibacterial soap.
If you're working in a daycare environment or are just changing another child's diaper then you should wear disposable gloves and you still need to wash both your hands and the child's hands after changing the diaper.