It usually takes more than just a bank account number for someone to be able to steal money from your bank account but it is possible if the person who got your bank account number knows what they're doing.
They could use your bank account number to go online and do a bank draft or ACH from your bank account to theirs so yes it's possible.
However whenever you write someone a check they can see your bank account number right on the bottom of the check so it's hard to keep your bank account number a secret.