Wrong question, right question
Since I'm very practically oriented, I feel like this question is a bit of a red herring. Heteroromantic asexuals make about 22% of the community, and about 13% of heteroromantics consider themselves part of the LGBT community.* People are arguing over a relatively small group, and in the mean time pushing away a much larger group who they nominally accept.
*Based on survey results which will be released very soon. Update: they have been released.
The real question is, are there spaces which asexuals and LGBT people can share? Should there be? Does this particular configuration help people? Language has an effect on this question. If an asexual thinks of themself as just an ally, they will be less likely to participate or share their own experiences. Is that what we want? If an asexual acts homophobic, or an LGBT person hates on aces, do we want to treat it as an attack from the outside, or as ignorance from the inside?
In my mind, the desired community organization should determine language, not the other way around. I've argued before that the asexual and queer communities benefit from shared spaces, and that's why asexuals are queer. However, there's no way for me to really know if the combined community works for every individual in all their individual circumstances and preferences. That's why asexuals are allowed to freely identify as queer or not. Let each person decide if it works for them.
Since the majority of heteroromantics don't identify as queer, that indicates that the queer identity does not work for them. I respect that. I also respect the minority who find the identity useful. If they identify as queer in the face of some obvious differences from the dominant queer subset (ie gay men), they must have a good reason for it.
Degrees of reclamation
One contention is that "queer" used to be a pejorative used against gay, lesbian, bisexual, and trans people. Therefore, only gay, lesbian, bisexual, and trans people have a right to reclaim the word.* I think of this as an extension of rules applied to many other reclaimed slurs. Gay men occasionally use "fag" to refer to each other, but it's not okay for others to do so. Black people may occasionally refer to each other as "nigga", but it's not okay for other people. Same for "dyke", "tranny", "slut" and numerous other words you can find on Wikipedia.
*See two examples of this argument in the wild
But if you look down the list, it's pretty clear that there are different stages of reclamation. "Nerd", "geek", "Quaker", and "Mormon" are also reclaimed words. There are no restrictions on these words because most of us can scarcely remember when they were slurs, and their reclaimed meaning has taken over their original meaning. At some point, the history of the term becomes irrelevant.
"Queer" is a term that is somewhere in the middle. I am certainly aware of its history as a slur, but I have never heard it used as a slur in my life. I am not suggesting that it never gets used pejoratively anymore, but that in at least a few geographical locations and environments, its positive meaning has eclipsed its previous meaning. The positive meaning that emerged is sort of a synonym for LGBT, with fewer syllables, and more of a liberationist connotation. It also tends to connote inclusion of people who fall into the cracks between boxes.
However, "queer", in its slur form, was not used to refer to all LGBT people. It was mainly used for effeminate men (or trans women) who were presumed to take a passive sexual position with other men. Does this mean that butch top men, lesbians, bisexual women, and trans afab people should not be using the word "queer"? (No it doesn't.)
Note that many people argue that trans men should not use "tranny", which is a slur mostly used against trans women. The fact that no one disputes trans men's use of "queer" serves to highlight the different natures of "queer" and "tranny". Here's a quote:
Reclaiming tr*nny feels like a way to have a history. But that word was never our [trans men's] history. It feels like a way to name and confront those invisible oppressive structures. But it doesn’t do that work, because while the structures that oppress trans women have many elements in common with the ones that oppress us, they’re not the exact same ones.By contrast, the asexual use of "queer" has nothing to with wanting to revise history, it has to do with the practical need for an umbrella term for sexual minorities. The use of "queer" as an umbrella term is exclusive to its reclaimed meaning, so I feel that using it this way can only serve to push its days as an insult further into the past. I don't think queer people really feel threatened by this usage. It's not like calling a woman "bitch", and then justifying it by saying, "But sometimes women call each other bitches!" I think they're mostly just bothered by the alleged relative degree of privilege asexuals enjoy over other queer groups.
You can identify as anything you want! But if it is absolutely imperative for you to use that word, and you using that word makes trans women feel unsafe around you, I’m not sure what to tell you. Maybe you should do some work within yourself, trying to discover why you have such an intense need to own a word that makes people feel unsafe.
The degree of marginalization experienced by asexuals is obviously a very broad topic which I will surely write about again and again. Here, I only mention it, since it's the main undercurrent of the "Asexuals are misappropriating the queer identity" argument. The claim is that asexuals aren't sufficiently systematically oppressed, so we have to take away their queer cards.
Arguments for this claim are basically arguments minimizing the issues that asexuals face. You can see why even non-hetero asexuals feel pushed away by the exercise, even though it's only supposed to apply to heteroromantic asexuals. I also feel disappointed on bisexuals' behalf whenever someone says asexuals "just" experience erasure. Bisexuals also experience erasure, and because of it, they're worse off than gay and lesbian folks. Clearly, open hate and discrimination aren't the only problems the LGBT community should be paying attention to, erasure is another.
Degrees of marginalization vary wildly depending on individual circumstances. I myself am immensely privileged, but since I am gay, no one tries to take away my queer card. I would venture that some heteroromantic asexuals, perhaps the 13% who identify with the LGBT community, are on the opposite end, experiencing marginalization that is within range of their local queer community. We don't really know how much marginalization individuals experience, and that's why we give people the freedom to choose for themselves.