Which statistical test should I use? A decision guide

Q: How do I choose a statistical test?

Answer three questions: what you're doing (comparing groups vs measuring a relationship), your data type (interval, ordinal, or categorical), and how many groups (and whether they're paired). Those determine the test — see the decision table above.

Q: When should I use a non-parametric test?

When your data is ordinal (like Likert scales), skewed, has outliers, or comes from a small sample where you can't verify normality. Use Mann–Whitney (2 groups), Wilcoxon signed-rank (paired), or Kruskal–Wallis (3+ groups).

Q: What's the difference between a t-test and ANOVA?

A t-test compares the averages of two groups; ANOVA compares three or more. Running many t-tests instead of one ANOVA inflates your chance of a false positive.

Q: Should I use a t-test or Mann–Whitney for Likert data?

Mann–Whitney. Likert data is ordinal and often non-normal, which violates the t-test's assumptions. Mann–Whitney is the correct non-parametric equivalent.

Q: What test do I use for yes/no or categorical data?

Chi-square test of independence, which checks whether two categorical variables are associated (e.g., does pricing page relate to whether users converted).

Q: Do I need to check assumptions before choosing?

Yes — parametric tests assume roughly normal, interval data. If those assumptions don't hold, switch to the non-parametric equivalent rather than forcing the parametric test.

Choosing the wrong statistical test is one of the most common — and most quietly damaging — mistakes in research analysis. Use a t-test where a non-parametric test belongs and your "significant" result may be invalid. This guide gives you a plain-English decision process for picking the right test, with examples, so your findings hold up.

Three questions that decide your test

Almost every test choice comes down to three questions:

What are you trying to do? Compare groups? Measure a relationship? Test an association between categories?
What type of data do you have? Interval/ratio (numbers with equal spacing, roughly normal) — or ordinal (ranked, like Likert) — or categorical (labels)?
How many groups, and are they related? Two groups or more? Independent groups, or the same people measured twice (paired)?

Answer those and the test picks itself.

The decision table

Goal	Data type	Groups	Test
Compare 2 group averages	Interval, normal	2 independent	Independent t-test
Compare before/after	Interval, normal	2 paired	Paired t-test
Compare 3+ group averages	Interval, normal	3+ independent	One-way ANOVA
Compare 2 groups	Ordinal / non-normal	2 independent	Mann–Whitney U
Compare before/after	Ordinal / non-normal	2 paired	Wilcoxon signed-rank
Compare 3+ groups	Ordinal / non-normal	3+ independent	Kruskal–Wallis
Test association between categories	Categorical	—	Chi-square
Measure strength of a relationship	Interval	2 variables	Correlation (Pearson)
Predict an outcome from predictors	Interval	1+ predictors	Regression (linear/multiple)

Parametric vs non-parametric (the key fork)

This is the fork most people get wrong. Parametric tests (t-test, ANOVA, Pearson) assume your data is roughly normally distributed and measured on an interval scale. Non-parametric tests (Mann–Whitney, Wilcoxon, Kruskal–Wallis) don't — they work on ranks and are the correct choice for:

Ordinal data — Likert scales, satisfaction ratings, rankings.
Skewed data — task times, income, anything with a long tail.
Small samples where you can't verify normality.
Data with outliers that would distort a mean.

Why this matters: running a t-test on 5-point Likert data is extremely common and often wrong, because Likert data is ordinal, not interval. The non-parametric equivalent (Mann–Whitney) is the safe, correct call. Most survey and UX-research tools don't even offer non-parametric tests — so people default to the wrong one.

Quick mapping of parametric → non-parametric equivalents:

Parametric	Non-parametric equivalent
Independent t-test	Mann–Whitney U
Paired t-test	Wilcoxon signed-rank
One-way ANOVA	Kruskal–Wallis

Worked examples

Example 1 — "Did satisfaction improve after the redesign?" Two independent groups (old vs new), satisfaction on a 1–7 Likert scale. Likert = ordinal → Mann–Whitney U. (Not a t-test.)

Example 2 — "Do users complete the task faster after onboarding changes?" Same users, before and after, task completion time. Time is skewed (long tail) and paired → Wilcoxon signed-rank. If times were normal, a paired t-test.

Example 3 — "Do three pricing pages convert differently?" Three groups, outcome = converted yes/no (categorical). → Chi-square test of independence.

Example 4 — "Do satisfaction scores differ across four user segments?" Four groups, satisfaction on an interval scale that's roughly normal → One-way ANOVA. If the scores were ordinal or skewed → Kruskal–Wallis.

Example 5 — "Does feature adoption predict retention?" Two continuous variables → correlation for strength; regression if you want a predictive model with coefficients (and to control for other factors).

What the output means

Every test returns a p-value: the probability the result happened by chance. p < 0.05 is the conventional threshold for "statistically significant." But always pair it with an effect size:

Cohen's d (t-tests), η² (ANOVA), r (correlation and non-parametric tests) — these tell you how big the effect is, not just whether it exists.
A huge sample can make a trivial difference "significant." Effect size keeps you honest.

A simple flow to follow

Comparing groups or measuring a relationship? Relationship → correlation/regression. Comparing → continue.
What's your outcome data? Categorical → chi-square. Ordinal/skewed → non-parametric branch. Interval/normal → parametric branch.
How many groups? Two → t-test or Mann–Whitney. Three+ → ANOVA or Kruskal–Wallis.
Same subjects measured twice? Yes → paired t-test or Wilcoxon signed-rank.

Common mistakes

Defaulting to a t-test for Likert data — it's ordinal; use Mann–Whitney or Wilcoxon.
Running multiple t-tests instead of ANOVA for 3+ groups — inflates your false-positive rate. Use ANOVA (or Kruskal–Wallis), then post-hoc comparisons.
Reporting significance without effect size — "p < 0.05" alone doesn't tell you if the effect matters.
Treating correlation as causation — a relationship isn't proof one thing causes the other.
Ignoring assumptions — parametric tests assume normality; check it, or go non-parametric.

Run the right test free

ResearchRocket's statistics engine covers the whole decision table — independent and paired t-tests, one-way ANOVA, chi-square, correlation, linear and multiple regression, and the non-parametric tests (Mann–Whitney, Wilcoxon, Kruskal–Wallis) — with plain-English significance readouts and effect sizes, run directly on your survey and study data. It auto-detects numeric vs grouping columns so you're guided to the right test. No SPSS, no exporting.

FAQ

How do I choose a statistical test? Answer three questions: what you're doing (comparing groups vs measuring a relationship), your data type (interval, ordinal, or categorical), and how many groups (and whether they're paired). Those determine the test — see the decision table above.

When should I use a non-parametric test? When your data is ordinal (like Likert scales), skewed, has outliers, or comes from a small sample where you can't verify normality. Use Mann–Whitney (2 groups), Wilcoxon signed-rank (paired), or Kruskal–Wallis (3+ groups).

What's the difference between a t-test and ANOVA? A t-test compares the averages of two groups; ANOVA compares three or more. Running many t-tests instead of one ANOVA inflates your chance of a false positive.

Should I use a t-test or Mann–Whitney for Likert data? Mann–Whitney. Likert data is ordinal and often non-normal, which violates the t-test's assumptions. Mann–Whitney is the correct non-parametric equivalent.

What test do I use for yes/no or categorical data? Chi-square test of independence, which checks whether two categorical variables are associated (e.g., does pricing page relate to whether users converted).

Do I need to check assumptions before choosing? Yes — parametric tests assume roughly normal, interval data. If those assumptions don't hold, switch to the non-parametric equivalent rather than forcing the parametric test.

Run the right statistical test free on ResearchRocket →