"Libertarians are selfish: they won't let me decide how their money is spent."
Not wanting to be extorted by the state or anyone else is hardly selfish. People avoid dark alleys and dangerous areas because they don't want to get mugged, and don't get accused of being selfish for not "contributing" to a robber's income. Rather, what is irrationally selfish is believing one has a claim to - deserves ownership of - another person's labor or goods.
In The Virtue of Selfishness, Ayn Rand writes about rational selfishness - "the values required for man's survival qua man" as a contrast to the "human sacrifice" demanded by statists, who desire the unearned, and demand sacrifices of others but do not make them. Instead, people ought to deal with one another voluntarily, as traders, without coercion or myths of false obligation (e.g., an alleged Social Contract).
Some statists have difficulty with the idea that in a free society, they would have to use persuasion rather than force to get people to give to their pet causes—some of which may be honestly benevolent, yet, nonetheless, the goodness of a cause does not create obligation nor justify force in its furtherance. In fact, bait and switch tactics are common here: while claims are made that the poor won't be educated or fed, typically people are generous in contributing to those actually in need (more so without the state's overhead and with competition for efficiency; see previous link), the concern tends to be more for less-necessary causes like bad art, owl preservation, or bailing out those that don't need it. (DR)
The selfishness charge is, of course, an ad hominem attack - not an argument. The non-aggression presumption implies that, by default, others should not steal my money. It is up to the statist to try to justify theft by State. Furthermore, many anarchists are egoist - either of the Ayn Rand or Max Stirner type. To those of us, being called "selfish" is a compliment. After all, as Ayn Rand said: Selfishness is a virtue. [HB]