Survey question design and bias review

You are a survey methodologist. Help me design a survey that produces reliable answers.

What I want to learn: [2-4 RESEARCH GOALS]
Respondents: [WHO, AND HOW THEY WILL RECEIVE IT, e.g. "customers by email, mostly on mobile"]
Maximum length: [e.g. "5 minutes", "12 questions"]
Decisions the results will inform: [e.g. "which feature to build next"]
Existing draft (optional):
"""
[PASTE DRAFT QUESTIONS, or "none"]
"""

If I gave a draft, review each question first:
- Flag problems: leading or loaded wording, two questions in one, vague terms ("often", "regularly"), jargon, overlapping or missing answer options, questions respondents cannot answer accurately.
- Give an improved version and one line explaining the change.

Then produce the final survey:
1. A one-sentence introduction that states purpose and time needed.
2. Questions in a logical order: easy and general first, sensitive or demographic questions last.
3. For each question: the type (single choice, multiple choice, rating scale, open text), the answer options, and which research goal it serves.
4. Use balanced rating scales with labelled points, and include "Not applicable" or "Prefer not to say" where needed.

Finish with: questions I could remove if the survey is too long, and a suggestion to test it with 3-5 people first.

Example (review excerpt):
Original: "How much do you love our new, faster checkout?"
Problem: leading ("love", "faster") and assumes the respondent used it.
Improved: "Have you used the new checkout?" (Yes / No / Not sure), then, if yes: "How easy or difficult was it to complete your purchase?" (Very difficult – Very easy, 5 points)

Original: "Is our app fast and reliable?"
Problem: two questions in one.
Improved: two separate rating questions, one on speed and one on reliability.

When to use it

Use this prompt before you send a customer survey, an employee pulse check, a course evaluation or a questionnaire for a research project. Small wording problems can skew results, and once a survey is out you cannot fix the data. The prompt is useful both for writing a survey from scratch and for checking a draft someone else wrote. Even a short internal poll benefits from a quick review, because people tend to trust survey numbers more than they deserve.

How to customize it

• [RESEARCH GOALS] should be specific. “Understand why trial users do not upgrade” leads to better questions than “get feedback”.
• [RESPONDENTS] and the delivery channel shape length and format. Mobile respondents tolerate fewer open-text questions.
• [MAXIMUM LENGTH] protects completion rates. Every question should earn its place.
• [DECISIONS] lets the model cut questions that are interesting but would not change anything.
• [EXISTING DRAFT] triggers the review step. Paste it exactly as respondents will see it.

Why it works

Survey methodology has well-known pitfalls: leading questions push respondents towards an answer, double-barrelled questions ask two things at once, vague frequency words mean different things to different people, and unbalanced scales skew results. Listing these explicitly gives the model a concrete checklist instead of a general instruction to “improve” the questions.

Linking each question to a research goal is a simple discipline that keeps surveys short. Ordering from easy to sensitive questions follows common practice for reducing drop-outs. The final suggestion to pilot with a few people matters most: a model can spot many wording problems, but only real respondents reveal what they actually misunderstand.

Tips and variations

• Ask the model to predict the three most likely ways respondents could misread each question, then use those predictions to guide your pilot test.
• Ask for the survey translated for each audience, then have native speakers check that the scale labels feel natural.
• For employee surveys, add “Make sure no question could identify a respondent in a small team.”
• After collecting responses, analyse open-text answers with our thematic analysis prompt.
• Our article on algorithmic bias shows how biased data collection can affect decisions later.