If you're the account holder for the checking or savings account then yes you can withdraw the money with just an account number.
But if the bank teller does not know you then they will ask for ID to show that you're the bank account holder.
If someone just goes up to the bank teller and asks to withdraw money and all they have is your bank account number then they will not be able to get money from your bank account.
Also to remove money or withdraw money from your bank account online you also need the routing number along with the account number.
It's not that easy to just withdraw someone's money from their bank account online though because the bank account holder has to give permission and approve the transfer of the money.
Your bank account number and routing number are on your checks so when you write a check to someone they see your bank account and routing number.
But there's not much they can do with it.