It is possible for someone to have their IP address hacked and then the hacker can get into a persons computer and router.
But hacking someone's IP address is not easy and is very difficult even for a seasoned hacker.
Your internet service provider has firewalls and routers that are very secure and they have teams of people working around the clock to prevent any hacking of their routers, firewalls and customers IP addresses.
It's also a good idea to have a firewall on your computer or router as well but your ISP has lots of very secure firewalls and experienced IT people that work to keep you from being hacked.
Whenever you visit a website your IP address is tracked and the website owner can see your public IP address which is needed to connect to the website.
Without your IP address and the IP address of the web server you would not be able to view websites or download anything from the internet.