Yes most babies do get a mild fever when they're teething but it doesn't always happen to every baby.
My nephew never got a fever while teething but his brother did so it's entirely dependent on the individual baby whether they'll get a mild fever or not while teething but most do get a fever when teething which is very normal.
Do note though that if the baby has a Fever higher than 101 F when they're teething then the fever is likely due to some other infection and not the teething itself.
The Fever should be very mild and not very high when the baby is teething.