I have a nearly 6 year old male cat that does the same. Although he will stay in the presence of my Parents and sister.
But whenever others are around he will hide away from them.
I guess it's just shyness in his part. Or he just doesn't like being around a lot of people.
That's okay though. We're both the same in that part. I don't like being around a lot of people either.
I like my alone time and prefer my own company. Well I do love my cat's company too.
Sometimes he'll go and sleep in the kitchen by himself and sometimes he'll sleep in my lap or even get in bed with me.