Without the other details such as routing number, other personal information or online bank account details then there's really not much the person can do should they get your bank account number.
Actually if you write checks if you look on the bottom of the check the number to your bank account is on that check so anyone you write a check too gets your checking account number.
But really there's nothing anyone can do to your bank account with just the bank account number.