If the police are at your door and do not have a warrant to enter and they knock and you don't answer there's nothing they can do.
However the police might be knocking on your door to let you know of something important so at least just open your front door and talk to the police through the screen or storm door.
I've had police come to my door to be aware that they were looking for an escaped dangerous prisoner and showed me a picture of the person.
So they could be knocking on the door to let you know of something for your safety.
If you've not been in trouble there's nothing to worry about.
I always just answer the door to see what they want and then they are on their way.
They could also be letting you know about a family member being injured or killed such as in a car crash.
So just answer the door and see what they want and if you did nothing wrong there's nothing to worry about.