Babies can safely go outside with you in any cold weather as long as you bundle the baby up and cover them up in the carrier.
However if the baby doesn't need to be out in the cold you should keep the baby inside the house where it's warm.
I would only take the baby out for a walk in a stroller when the temperature is 35 F or above as long as you have the baby in some warm clothes and a blanket to keep the baby warm.
However don't stay out too long to avoid the baby getting too cold.