Touching worms does not hurt the worms.
Also there's no harm to yourself when touching worms either.
I used to pick worms up as a kid bare handed and they never were harmed by doing so.
Just be sure to wash your hands after you touch the worms before eating food or touching food.
There's no need to use gloves to touch worms even when using worms as bait for fishing.
I've always just picked up the worms with my bare hands and put the worms on the hook.
I just made sure to wash my hands after touching them before I ate food or prepared food.