It depends on the jail and the cell block you are in.
Some jail cells do have windows in them while other jail cells don't.
If they do have windows they have really small windows that are unbreakable and even if you did manage to somehow break the window. The window is so small you would not be able to squeeze through the window.