I disagree. You have a very "occidental centered" PoV, potentially ignoring hundred of millions of Chinese and Indians men, which are certainly as ableist as white men.
You live in a white men dominated "world", this is for sure (and for the worst for now) but it is not the case, for example, for the ~60 millions people living with an handicap in India.
Wenjie, born blind in Datong - China, don't live in a white men dominated world but certainly face discrimination every day. The same for Nithya, who has a palsy and live in Hyderabad. Both are fictional characters but you certainly get my point.
@beanface42 but @Rhube is talking about her experience. It's not men of color who come into her mentions, I guess. Although I admit I made a broad statement. Sexism is present all over the globe with specifities in each culture, that's right.