Surveys and validation

This has been cross-posted on The Asexual Agenda.  Obligatory disclaimer: My views do not necessarily reflect those of the AVEN survey committee. I speak only for myself.

When the AVEN Community Census, a volunteer-run survey of asexual communities, approached the data analysis phase, someone on the committee said, "Who will help me eat the bread?" That phrase stuck with me as a particularly apt description of survey creation, albeit completely backwards.

The question, "Who will help me eat the bread?" comes from the folk story, "The Little Red Hen". In the story, the hen asks for volunteers to help in the various steps to create bread, but none will volunteer. Finally, she asks for volunteers to eat the bread, and suddenly everyone is feeling helpful!

As the committee member put it, writing the questions and disseminating the survey is the hard work to make bread. Analyzing the survey is eating the bread. Everyone wants to know the results!

But having actually worked on the survey analysis, my impression is that writing the questions is eating the bread, while data analysis is just plain work. Even when it comes to myself, I'm enthusiastic about doing analysis but it's a lot of work and I can't seem to find the time. (In contrast, finding time for blogging is much easier.) And this is not a singular pattern. You can see from the history of asexual community surveys that the hardest part is getting the analysis moving.

Writing questions also garners a lot of community interest, judging by the abundance of complaints about questions, and scarcity of complaints about the lack of analysis. More specifically, people are interested in what's in the questions, because questions are a form of validation. People really want to see their particular identity, their particular experience, and their particular views reflected in the survey.

And this is all wrong, in my opinion. What makes a good survey question is independent of validating identities. For example, when choosing what romantic orientations appear on the survey, the primary consideration is to capture the most common combinations of identities, basically so we can minimize the labor of interpreting write-in responses, and create a succinct summary of results. To assign validity according to whether the romantic orientation appears in the survey is to assign validity based on popularity.

There are other concerns too, such as whether an experience can be easily quantified, whether it tells us something we already knew, and whether it's the sort of thing we want to measure on an annual basis, or just for one year. Plus there are a bunch of random personal factors and plain stupid oversights, so I'm not saying it's wrong for people to complain.

Complaints about survey questions evoked many mixed feelings in me.  Some critiques of the survey were valid, and I don't wish to discourage people from giving them.  And even if there's nothing to be changed, people's feelings are still valid and can be expressed freely.  But clearly a lot of people have an emotional need for validation from authority, any kind of authority at all. I don't want to be your benevolent authority, I want you to be free.