If you do not wash off soap from your skin then your skin will start drying out and then your skin will become itchy and you will likely even develop a rash on your skin.
I forgot to wash some soap off my back once and it became really itchy and I started getting a rash and I had remembered that the reason was that I forgot to rinse the soap off my back after I used a back scrubbing brush in the shower.
I thought I had washed the soap off my back but I actually forgot to rinse the soap off and it was really itchy.
So make sure you always wash the soap off your skin after applying it because you can get rashes as well as dry skin and itchy skin.