How to analyze survey data (a practical guide with examples)

Collecting survey responses is easy. Turning them into a defensible finding — one that survives a "how do you know that's not just noise?" from a stakeholder — is where most survey analysis falls apart. This guide walks through analyzing survey data properly: descriptive statistics, the significance tests that separate signal from noise, which test to use when, and how to handle open-ended responses. With worked examples throughout.

The two halves of survey analysis

Every survey has two kinds of data, and they need different treatment:

• Closed-ended (quantitative) — ratings, scales, multiple choice, NPS/CSAT. Analyzed with statistics.
• Open-ended (qualitative) — free-text comments. Analyzed with thematic coding.

The best analyses combine them: the numbers tell you what is happening, the comments tell you why.

Step 1: Clean the data first

Before any analysis:

• Remove incompletes and speeders — respondents who finished implausibly fast or skipped most questions.
• Check for straight-lining — someone who answered "5" to every Likert item isn't giving real data.
• Handle missing values deliberately — decide whether to exclude a respondent or a question, and be consistent.
• Recode where needed — reverse-scored items, grouped age bands, etc.

Garbage in, garbage out applies doubly to statistics — a significant result on dirty data is worse than no result.

Step 2: Start with descriptive statistics

Descriptives summarize each question before you test anything:

• Mean, median, mode — central tendency. For skewed data (income, task times), the median is more honest than the mean.
• Standard deviation — how spread out responses are. High SD means people disagree; that's often the real story.
• Frequencies and percentages — for categorical questions.
• 95% confidence interval — the range the true value likely sits in, given your sample.

Worked example. You survey 120 users on a 1–7 satisfaction scale after a redesign. Mean = 5.4, median = 6, SD = 1.3, 95% CI [5.17, 5.63]. The CI tells you the true average satisfaction is very likely between 5.2 and 5.6 — useful context a bare "5.4" hides.

Step 3: Ask whether differences are real (significance testing)

The moment you compare groups — "did satisfaction go up after the redesign?", "do power users rate us higher than new users?" — you need a significance test. It answers one question: is this difference likely real, or could it be random chance? The output is a p-value; by convention, p < 0.05 means the result is statistically significant (less than a 5% chance it's noise).

Which test you use depends on your data. Here's the decision guide (a fuller version lives in which statistical test should I use):

Why non-parametric matters for surveys: Likert and rating-scale data are ordinal and rarely normally distributed. Running a t-test on a 5-point Likert scale is a common mistake. For ordinal data, the non-parametric tests (Mann–Whitney, Wilcoxon, Kruskal–Wallis) are the statistically correct choice — and most survey tools don't offer them at all.

Step 4: A worked significance example

Back to the redesign. You want to know: did satisfaction actually improve, or does it just look higher? You have satisfaction scores from 60 users on the old design and 60 on the new one, both on a 1–7 scale.

Because the scale is ordinal, you run a Mann–Whitney U test:

• Old design: median 4.5 · New design: median 6.0
• U = 1,120, z = −3.42, p < 0.001
• Effect size r = 0.31 (a moderate effect)

Interpretation: the improvement is highly statistically significant (p < 0.001 — far below 0.05), so it's very unlikely to be chance, and the effect size (0.31) says it's a moderate, meaningful shift — not just technically significant. You can tell stakeholders the redesign improved satisfaction with confidence, and quantify how much.

Contrast: if that same comparison came back p = 0.18, you'd report "no significant difference" — the scores moved, but not enough to rule out chance. Reporting that honestly is just as valuable; it stops the team shipping a change that didn't actually help.

Step 5: Look for relationships (correlation and regression)

To ask "does X predict Y?" — e.g., does time-in-product predict satisfaction? — use correlation (strength of a relationship) or regression (a predictive model with coefficients).

Example. A regression of satisfaction on three predictors (sessions/week, features used, support tickets) might return: features used (β = 0.42, p < 0.01) and support tickets (β = −0.30, p = 0.03) are significant; sessions/week is not (p = 0.21). Read: breadth of feature use drives satisfaction up, support contact drives it down, and raw session count doesn't matter once you account for the other two. That's a roadmap insight, not a vanity chart.

Step 6: Analyze the open-ended responses

Don't skip the free-text. Code the comments into themes (thematic analysis) and count how many respondents raised each. Then cross-reference with the numbers: if satisfaction is up but a "billing confusion" theme keeps recurring, you've found your next fix. AI auto-coding makes this fast even for hundreds of responses.

Step 7: Report responsibly

• Report the p-value and the effect size — significance says "it's real," effect size says "how much it matters." A tiny effect can be significant with a big sample and still be irrelevant.
• Never claim causation from a correlation without a controlled design.
• State your sample size and confidence level.
• Show the honest nulls — "no significant difference" is a finding, not a failure.

Common survey-analysis mistakes

• Running a t-test on Likert/ordinal data instead of a non-parametric test.
• Reporting means without spread or confidence intervals — hides disagreement and uncertainty.
• P-hacking — testing everything until something hits p < 0.05, then reporting only that.
• Confusing significance with importance — a large sample makes trivial differences "significant."
• Ignoring the open-ended data — the numbers say what; the comments say why.

Analyze survey data free

ResearchRocket runs surveys and analyzes them in one place, with a real statistics engine: descriptives with confidence intervals, independent and paired t-tests, one-way ANOVA, chi-square, correlation, linear and multiple regression, and the non-parametric tests (Mann–Whitney, Wilcoxon, Kruskal–Wallis) that ordinal survey data actually requires — plus AI thematic coding for the open-ended responses. No exporting to SPSS. None of the major UX-research competitors offer real significance testing.

FAQ

How do you analyze survey data? Clean the data, run descriptive statistics (mean, median, SD, confidence intervals), use significance tests to check whether group differences are real (choosing the right test for your data type), analyze open-ended responses with thematic coding, and report p-values alongside effect sizes.

Which statistical test should I use for survey data? For comparing two groups on normal data, a t-test; for ordinal/Likert data, Mann–Whitney U; for 3+ groups, ANOVA (normal) or Kruskal–Wallis (ordinal); for before/after, a paired t-test or Wilcoxon signed-rank; for categories, chi-square. See the full decision guide.

Can I run a t-test on Likert scale data? It's generally not recommended — Likert data is ordinal and often non-normal, so a non-parametric test (Mann–Whitney or Wilcoxon) is the statistically correct choice. This is one of the most common survey-analysis mistakes.

What is a p-value in survey analysis? The probability that a difference you observed happened by chance. By convention, p < 0.05 means the result is statistically significant. Always report it alongside an effect size, which tells you how large the difference is.

Do I need SPSS to analyze survey data? No. Tools like ResearchRocket run the same core tests (t-tests, ANOVA, regression, non-parametric) directly on your survey results, so you don't have to export to SPSS, R, or Excel.

How do I analyze open-ended survey responses? Code them into themes (thematic analysis) and count how many respondents raised each, then cross-reference with your quantitative results. AI auto-coding speeds up the first pass for large volumes.

Analyze survey data free on ResearchRocket →

Related: Which statistical test should I use? · How to design a survey · Free sample-size calculator · How to do thematic analysis