Questionnaire validity and reliability: a practical guide for postgraduate researchers

Almost every methodology chapter in a master's or doctoral thesis has a section called "validity and reliability of the instrument". Yet many researchers mix the two up, or report a Cronbach's alpha and treat it as proof of both. This guide explains the difference, the main types of each, and what an examining committee expects you to show.

Validity and reliability are not the same {#difference}

• Validity: does the questionnaire measure what it was built to measure? A "job satisfaction" questionnaire is valid when its items really measure job satisfaction, not loyalty to a manager or a wish to please the researcher.
• Reliability: does the questionnaire give consistent results? If it were given again to the same people under similar conditions, or if its items were compared with each other, would the picture be similar?

The usual picture is a target. A reliable archer puts the arrows close together, but possibly far from the centre. A valid archer hits the centre. Reliability is necessary for validity, but not enough on its own: a questionnaire can be very consistent about the wrong thing.

That is why "Cronbach's alpha was 0.89, which shows the instrument is valid and reliable" is not a sound sentence. Alpha tells you about reliability only.

Types of validity {#validity-types}

Face and content validity {#content}

Face validity means the items look relevant and clear to someone reading them. Content validity goes further: do the items cover every part of the concept you are measuring, without straying beyond it?

The usual method is expert review: the draft is sent to several subject experts, who judge each item on whether it belongs to its construct, how clearly it is worded and whether the language is correct, and suggest items to drop, add or change. The questionnaire is then revised along the lines most of them agree on.

Some universities ask for this agreement as a number, such as Lawshe's content validity ratio (CVR) or the content validity index (CVI). Ask your supervisor which one your university uses.

Important content validity is an expert judgement. No statistics program can compute it from respondents' answers.

Internal consistency validity {#internal-consistency}

In many theses this means the correlation of each item with the total score of its construct, and of each construct with the total score of the questionnaire. An item whose correlation is weak, or not significant, is a candidate for removal or rewording.

The more precise figure is the corrected item-total correlation: the item's correlation with the sum of the other items in its construct. Including the item in the total it is compared with inflates the correlation, especially in short constructs. Many researchers treat a value below 0.30 as a sign of a weak item. It is a rule of thumb, not a law.

Construct validity {#construct}

Construct validity asks: do the items group into the constructs you designed? If you say your questionnaire measures three dimensions, do the data actually show three?

It is usually examined with factor analysis, covered below. It also includes:

• Convergent validity: your instrument correlates strongly with another instrument that measures the same concept.
• Discriminant validity: it correlates weakly with an instrument that measures a different concept.

Criterion validity {#criterion}

The questionnaire's scores relate to a known outside criterion. For example: do students who score higher on an "academic engagement" scale have higher grades? When the criterion is measured at the same time it is called concurrent validity; when it is measured later, predictive validity.

How reliability is measured {#reliability-methods}

Two rules apply to every method:

1. Compute reliability for each construct, not only for the whole questionnaire. A questionnaire with five different constructs can show a high alpha overall even when one construct is weak.
2. Reverse the negatively worded items first. An item such as "I miss lectures without a good reason" in an engagement scale must be reverse scored, or reliability drops when nothing is really wrong with the instrument.

Exploratory factor analysis in brief {#efa}

Exploratory factor analysis (EFA) looks for groups of items that correlate strongly with each other, and calls each group a "factor". If your design is sound, the factors match your constructs.

What a thesis usually reports:

1. Whether the data suit the analysis: the KMO measure, where 0.60 or more is usually considered acceptable and closer to 1 is better, and Bartlett's test of sphericity, which should be significant.
2. How many factors: eigenvalues above 1, the scree plot, and the variance explained.
3. Loadings after rotation: how strongly each item loads on its factor. Many researchers use 0.40 as a minimum, and look again at any item that loads similarly on two factors.

Some studies then run confirmatory factor analysis (CFA), which tests a model specified in advance with fit indices such as CFI and RMSEA. It is usually done in dedicated software such as AMOS or the lavaan package in R.

Note factor analysis needs an adequate sample. Researchers quote rules such as 5 to 10 respondents per item, or at least 200 respondents, but these are rough guides that depend on how strong the loadings are. See our article on sample size.

How to write it up {#write-up}

Order the section in your methodology chapter like this:

1. Face and content validity: how many experts and in which fields, what they judged, and what changed after review (items removed and reworded).
2. The pilot study: its size, where it came from, and that it was outside the main sample.
3. Internal consistency: a table of each item's correlation with its construct.
4. Construct validity, if you ran it: KMO, Bartlett, the number of factors, the variance explained and the loadings table.
5. Reliability: a table with the alpha of each construct and of the whole instrument, with the number of items in each.

An example sentence:

"To establish reliability, Cronbach's alpha was computed for each construct. Values ranged from 0.81 to 0.92, and 0.94 for the instrument as a whole, indicating high internal consistency."

The figures in this example are illustrative.

Common mistakes {#mistakes}

• Using alpha to claim validity. Alpha measures reliability only.
• Computing alpha for the whole questionnaire but not its constructs. A weak construct disappears inside the total.
• Forgetting to reverse the negative items. Reliability drops and some correlations flip sign.
• Taking the pilot from the main sample and then including those answers in the final analysis.
• Running factor analysis on a very small sample and building conclusions on it.
• Translating a published scale without checking its validity in the new setting. A published scale needs its author's permission, and a translation needs expert review and a pilot.

Where Pulseform helps {#pulseform}

When you build your questionnaire as an academic survey, set a construct for each item and mark the reverse scored items, Pulseform computes from the answers directly:

• Cronbach's alpha for each construct with a clear verdict, alpha if item deleted, and the corrected item-total correlation,
• Exploratory factor analysis: KMO, Bartlett, eigenvalues and loadings after Varimax rotation,
• A correlation matrix between constructs, recommending Pearson or Spearman depending on normality.

Confirmatory factor analysis is deliberately left out, and you can export your data to SPSS to run it. Content validity through expert review remains your work and your reviewers'. The details are in the academic surveys guide, and a fully worked example with numbers is in Read a report with me.

References {#sources}

The original sources of the formulas, thresholds and rules of thumb in this article:

• Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334.
• Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2019). Multivariate data analysis (8th ed.). Cengage Learning.
• Kaiser, H. F. (1974). An index of factorial simplicity. Psychometrika, 39(1), 31–36.
• Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575.
• MacCallum, R. C., Widaman, K. F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4(1), 84–99.
• McDonald, R. P. (1999). Test theory: A unified treatment. Lawrence Erlbaum Associates.
• Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill.
• Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what's being reported? Critique and recommendations. Research in Nursing & Health, 29(5), 489–497.