Reacting to that recent college privilege exercise, Megan McArdle gets it right:
To me, privilege is not about how enjoyable your parents were able to make your childhood leisure time. It's not even about material goods....
Privilege describes how much scope your parents bequeathed you to shape your destiny. This operates in multiple and often subtle ways. It can be reading in the home, or a peer group carefully selected (usually through real estate purchase) to ensure that you "choose" to go to a competitive college instead of dropping out of high school and selling drugs. Or it might be the way having affluent, stable families enables people like me to opt for high-status, low-paying, personally enriching careers, because we know that if something really awful happens, our families can help out.
That's not the only kind of privilege (unearned advantage), of course: there's the unearned advantage of the able-bodied, and being the right skin color or sex, and even being born in the right place.