Yes you can give your baby a lukewarm bath or sponge bath when your baby has a fever.
Also giving your baby a bath while he/she has a fever may also help reduce the babies fever and make him/her feel better.
Bathing a baby or child with a fever was very common as a way of reducing fevers before fever reducing medicine became available.