You're just being paranoid.
If they don't know your IP address, it makes it hard or impossible to hack you.
But just knowing the IP address is not enough.
You reveal your ip address to every site you visit so there's no real hiding your ip address unless you use a proxy.