Some months ago, there was a comment scuffle on my other blog. One person, let's call them Alice, complains about transphobia and ablism in a particular forum. Another person, let's call them Bob, says that the allegations are too vague. Bob asks for a link and talks about recollection bias. Alice responds angrily, and among other things accuses Bob of being part of a privileged group. Bob asserts that this is ad hominem, and confesses dislike for the very idea of "privilege".
Bob is likely not alone in this perception, that the main function of "privilege" is to discount people's opinions based on who they are. When a privileged person says something an underprivileged person doesn't like, they can claim that the privileged person's opinion comes from privilege, and should therefore be discounted. This allows you to reject other people's views independently of their content, and thus it cannot help you achieve a greater approximation of truth.
It's hard to argue with that, if "privilege" is just used to indiscriminately dismiss people that you disagree with.
But there's also a way to translate "privilege" into skeptical terms. When someone is called out on their privilege, the translation is that they've been accused of bias. The paradigmatic case is when a white person expresses their personal impression that racism isn't a big deal these days. Of course, impressions come from personal experiences, and it hardly needs saying that people have different personal experiences. People also suffer from inattentive blindness, and thus are unlikely to notice comments or actions which don't hurt them personally.
That said, accusing people of privilege-induced blindness is a pretty shitty argument, because accusing people of cognitive bias is a shitty argument. Or at least, it's very hard to make persuasive case out of it.
For example, suppose someone believes in chemtrail-related conspiracies, and your response is to talk about the systematic bias which causes people perceive agenticity where there is none. It may be interesting to discuss, especially to third parties, but your debate opponent will likely remain unconvinced. For one thing, simply asserting a cognitive bias doesn't mean that it's there. Why should your opponent accept your assertion just on your say so? And for another, just because they have a statistically higher probability of having a certain set of false beliefs does not mean that this particular belief is false.
Another example: suppose someone believes that a particular forum is terrible, and your response is to talk about recollection bias. You're obviously not going to convince someone that their memory is wrong just because you've cited the fact that memories can be wrong.
(In general, I support the idea of privilege, but at some point I'd also like to enumerate its many problems and failings. Also at some point I'd like to discuss the failings of "name that fallacy" style argumentation, but my thoughts aren't fully formed.)