Yes you can become sick from breathing in cat litter dust but what can make you sick is the Silica dust that cat litter contains.
If your cat goes outside often then there's also the risk of inhaling the dust and getting Toxoplasmosis from your cats litter if your cat does become infected with it.
So although you can get sick from breathing in cat litter it's usually pretty rare to get sick from it.
But still it's a good idea to wear a dust mask when working around your cats litter box such as changing the cat litter or scooping the cats poop from the cat litter.