There's nothing wrong at all with wearing your Pajamas in public.
It's just that some people are offended by it because they see it as you not being fully dressed for society.
Pajamas are supposed to be only worn around the house or while sleeping to make you more comfortable but some people in society expect you to be fully dressed in proper clothing.
In my opinion when you're wearing Pajamas in public you're fully clothed so there should be no issue.
So if you want to wear Pajamas in public then just ignore the public that says it's wrong because that's just their opinion and you're not doing anything wrong wearing Pajamas anytime you want.