If you're up already at night then it's okay to take the puppy outside after waking it up to pee and then bring the puppy back in to sleep.
However if the puppy is asleep and you're sleeping yourself then there's no need to wake the puppy up to take them out to pee.
But what I would do is just remove the water bowl an hour before it's time for bed that way the puppy will not drink anymore water until the next morning.
Then the puppy won't have a full bladder and need to pee.
Once you get the puppy awake you may have a hard time getting the puppy back to sleep again sorta like waking a kid at night to change their diaper and then getting the kid back to sleep.