The two colors that should not be worn together are black and brown.
Black and brown colors should not be worn together as historically wearing black and brown colors together was viewed as clashing darks, although mixing black and brown colors together carefully with proper contrast and texture is actually now popular.
Other colors that should not be worn together include red and green, navy blue and black, brown and gray and green and orange.
Green and Orange colors are competing bold hues that can overwhelm the eye or even clash aggressively.
Brown and Gray are both muted neutral tones that can make an entire outfit look dull, flat, or even washed out.
Navy Blue and Black can look like an accidental mismatch or even a faded outfit because the two dark shades are so similar.
Red and Green are often avoided outside of the holiday season because they strongly evoke Christmas.
However if you're not into fashion, you can wear any colors together that you want to wear as there's no actual rule.
Also traditional fashion "rules" warn against pairing certain color combinations together, although modern style often breaks these fashion guidelines.