The reason there's a stigma for adults wearing diapers is because diapers are associated with babies.
People accept that babies and toddlers need to wear diapers and consider it normal and not shameful but as an adult we are expected by society to use the toilet and not need diapers.
However diapers are for all ages and if an adult needs to wear diapers for incontinence then they should wear the diaper.
Nobody will know you're wearing an adult diaper anyway if you need to or want to wear the diaper as long as you're wearing pants or shorts over the diaper.
Sweat Pants though tend to show the diaper bulge more.
I can see the diaper bulge in toddlers pants when they are wearing sweat pants so people would be able to tell you're wearing a diaper through sweat pants.
My own 3 year old boy who is still in diapers will wear sweat pants most days because it's easier to change his diaper that way.
I can easily tell he has the diaper on because it shows through his sweat pants in the back especially.
When he's wearing blue jeans I can hardly tell he has a diaper on even though I know it.