When you have astigmatism you don't always have to wear the eyeglasses or contact lenses if you do not want to or when you can see somewhat and don't need the eyeglasses or contact lenses.
However if you want to see properly and when you need to see properly you should wear the eyeglasses during those times or wear your contact lenses.
If you want to completely stop wearing eyeglasses or contact lenses then you could consider getting Lasik or laser eye surgery which can cure the astigmatism.
But not everyone is a good fit for Lasik or Laser eye surgery and it does cost quite a bit to have laser or Lasik eye surgery but can be worth it to improve your eyesight from the astigmatism.