A person cannot go to the bank and withdraw money from your bank account with the routing number and account number.
However they can go online and do an ACH transfer of money to take money out of your account that way.
It's also known as eCheck payments so yes if they have both the routing number and checking account number then money can be removed from your checking account.