There's nothing wrong at all with not spending time with some family that you don't like being around.
I have family members myself that I can't stand to be around and they're all the time wanting me to show up and visit them in other states.
I refuse to do so because I don't like them and I've told them that but they say that we should see each other and spend time together because we are family.
That is wrong and just because you're family doesn't mean that you have to visit them or spend time with them.
So if you don't want to spend time with some of your family then that's okay and only spend time with the family that you want to spend time with.
Time is very short on earth so you should not waste your time with the people you don't like to spend time with.