Doctors touch a mans balls during a physical to feel for any lumps that could mean testicular cancer.
So when your doctor touches your balls they are examining your balls to test for cancer of your testicles.
When the doctor touches your balls during the physical the doctor will grasp one testicle at a time, rolling it gently between the thumb and first finger to feel for lumps.
It can feel weird to have your balls touched by the doctor but to the doctor it's no big deal and just part of the doctors job.
The doctor has touched many other men's balls during a physical and it's no big deal to them so you shouldn't feel embarrassed or need to feel embarrassed.