How to design a master's thesis questionnaire from scratch

Many problems in a results chapter start before a single answer is collected: a construct that doesn't connect to the research question, an item that can be read two ways, a demographic question collected and never used, or one you needed in the analysis and never asked. This article walks through questionnaire design in the order that avoids them.

Before you start: if a published, validated scale already measures what you need, use it rather than building a new instrument, with its author's permission and adapted to your setting if needed. Building an instrument from scratch is a large piece of work, and its validity and reliability have to be established in full.

1. From research question to constructs {#constructs}

Start from your research questions and hypotheses, not from items. For each question ask: what concept do I need to measure to answer it?

Example: the question "How is academic engagement related to achievement among university students?" needs you to measure:

• Academic engagement, which the literature tells you has three dimensions: behavioural, cognitive and emotional,
• Achievement, which can be the GPA as a single numeric question.

Each dimension becomes a construct in the questionnaire. Then draw a simple table linking each research question to the constructs that answer it, and to the statistical test you will use. A construct that doesn't appear in that table has no place in the questionnaire.

How many items per construct? Four to six at least is the usual advice: a construct with two or three items struggles to reach good reliability, and a single-item construct has no internal consistency to compute at all. A very long questionnaire tires respondents, and answers towards the end get worse.

2. Writing clear items {#wording}

• One idea per item. "The lecturer is knowledgeable and helpful" is two items.
• The respondent's language. Write for a student, an employee or a parent, not for a fellow researcher.
• Short and specific. "I attend lectures on time" beats "I generally make an effort to observe the timing of various academic activities".
• No leading. Avoid "Don't you think that..." and "It is important that...".
• No double negatives.
• One or two reverse items per construct, worded in the opposite direction and reversed before analysis. The details are in our article on the Likert scale.

Then choose the answer scale, usually a 5-point Likert scale, and keep it the same throughout the questionnaire where you can: changing scales between constructs confuses respondents and makes comparison harder.

3. Demographics {#demographics}

Collect only what you will use in the analysis. If your hypotheses are about differences by gender, year of study and major, those are what you need. A question about income that will never appear in a table makes the questionnaire longer and costs you respondents' trust.

For each demographic question:

• Mutually exclusive and exhaustive categories: an age of 30 doesn't fall in two bands, and everyone has a band to choose.
• Categories that aren't too small: comparing a group of three people with a group of two hundred gives no result you can rely on. Merge small categories before collecting if you can.
• Placement: at the start if they are short and not sensitive, at the end if they are sensitive.

4. Ordering the questionnaire {#structure}

1. A short introduction: who you are, the topic, how long it takes, that taking part is voluntary, and that answers are confidential and used for research only.
2. Demographics, if you decided to put them first.
3. The constructs, each with a clear heading, or without one if a heading would lead the answers.
4. A thank-you at the end.

5. Expert review {#expert-review}

Send the draft to experts in your subject and in measurement; many universities ask for five to ten. Send them, with the questionnaire, the study's title, its research questions and a definition of each construct, and ask them to judge each item on whether it belongs to its construct, how clear it is and whether the language is correct, with space for a suggested change.

Then revise along the lines most of them agree on, and keep a record of what you removed, changed and added, because you will report it in the methodology chapter. The details are in our article on questionnaire validity and reliability.

6. The pilot study {#pilot}

Before the full rollout, give the questionnaire to a small sample from your population outside the main sample; around 30 respondents is the size researchers commonly quote. It lets you:

• check the items are clear: ask a few participants what they understood by the harder items,
• time the questionnaire,
• compute an early reliability for each construct and catch weak items while there is still time,
• test the collection method itself: the link, the devices or the paper.

Pilot answers do not go into the final analysis.

7. Approvals and ethics {#ethics}

Most universities require research ethics committee approval before collection, and the approval of the institution you collect from (a school, university or employer). Write enough in the introduction for the decision to take part to be an informed one. When respondents are minors, you will normally need parental consent.

8. Publishing and collecting {#collect}

• Set the sample size before you start, and write down how you chose it. See our article on sample size.
• Choose the channel that reaches your population: an online link, field collection on devices, or paper.
• Track the number of responses during collection, and how they spread across the groups you will compare.
• Don't change items once collection has started. Any change makes old and new answers incomparable.

In Pulseform {#pulseform}

• When you create a survey with the Academic & Postgraduate Research purpose, there is a ready-made survey for each common study design: correlational, comparative, descriptive, pre/post and instrument validation. Each has 4 to 6 items per construct and a reverse-worded item already marked, so you adapt it instead of starting from nothing.
• Every item has a Construct / Study Dimension field and a This item is reverse scored switch. In an academic survey, rating questions are always required, so missing answers can't distort reliability and construct means.
• In Settings → Analysis & Tests you choose your study design, and the right tests are switched on.
• You publish and share the survey with a public link and QR code, and you can collect the pilot in a separate survey so it never mixes with the main sample.

The details are in Create a survey, Survey settings and Academic surveys.

References {#sources}

The original sources of the formulas, thresholds and rules of thumb in this article:

• DeVellis, R. F. (2017). Scale development: Theory and applications (4th ed.). SAGE Publications.
• Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). John Wiley & Sons.
• Johanson, G. A., & Brooks, G. P. (2010). Initial scale development: Sample size for pilot studies. Educational and Psychological Measurement, 70(3), 394–400.
• Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575.
• Preston, C. C., & Colman, A. M. (2000). Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent preferences. Acta Psychologica, 104(1), 1–15.