If someone has your bank account number as well as other bank account details and your login details to an online account then yes they can easily withdraw and transfer money out of your banking account into their own bank account.
But if they try to go to the bank with your account details in person then that person is unlikely to be able to withdraw the money since the bank should ask for ID of the person withdrawing money.
So if you're trying to withdraw money from someone's bank account even with their permission then it's unlikely that the bank will let you withdraw the money.
That is unless you're listed on the bank account with permission to withdraw the money from the bank account or you're a power of attorney for that person.
Now if someone gives you their debit card and wants you to withdraw some cash from the ATM then yes as long as you have that persons permission along with their debit card and pin number then yes you can withdraw some cash from their account but only if they gave you permission to do so.