Yes your teeth can and will move or shift on you if you do not wear your retainer for one day or one night.
So if you just got your retainer in you should always wear the retainer until you no longer need to wear it or your teeth will shift out of position and then it will defeat the purpose of wearing the retainer in the first place.
I made the mistake of removing my retainer in my teen years because I was not comfortable one night wearing it so I removed the retainer.
Then the next day the retainer didn't want to fit correctly like it used to before.
I went to the dentist and the dentist said that my teeth had shifted as a result of removing the retainer and told me I should not remove the retainer even at night.
So always keep your retainer in your mouth to keep your teeth from shifting.