When Two Numbers Tell Different Stories: Making Sense of Survey Results
Intro
Imagine you run a neighbourhood café and want to introduce a new lunch offer. You have two adverts ready, but only enough budget to promote one.
You show Advert A to 20 potential customers and Advert B to another 20. Twelve people who see A say they would consider visiting. Ten people who see B say the same.
That is 60% for Advert A and 50% for Advert B.
A looks like the winner. But before you spend your advertising budget, you pause. The lead comes down to just two people. If you had asked different customers, would A still have come out ahead?
That question sits behind many everyday research decisions. A customer satisfaction score increases. One product concept receives more interest. A brand looks stronger among one audience than another. The difference is easy to see; deciding how much to trust it takes more thought.
I built two free tools to help with that decision. One checks a single comparison between two percentages. The other checks a set of comparisons together. To understand when each is useful, let's return to the café.
Why a higher percentage is only the beginning
Your café has more than 40 potential customers. The people you asked are a sample, a smaller group whose answers you use to learn about a wider audience.
Different samples will give slightly different answers. You might happen to speak to more enthusiastic people in one group, even if the two adverts are equally appealing overall. Researchers call this sampling variation.
This is why the percentage alone is not enough. You also need to know how many people it represents.
A result of 60% could mean 12 out of 20 people or 120 out of 200. Both give the same percentage, but the larger sample generally provides a more precise picture, assuming the people were selected appropriately.
A bigger sample cannot fix a poorly worded question or an unrepresentative audience. It does, however, help reduce uncertainty caused by the limited number of responses.
The method behind the tools, in plain language
Both tools use a method called a two-proportion Z-test. The name sounds more complicated than the question it answers.
A proportion is simply the share of people who give a particular response: the percentage who would visit the café, recommend a business or consider buying a product.
The test starts by assuming there is no underlying difference between the two groups. It then asks: How unusual would a gap this large be if that assumption were true?
To answer, it considers both the size of the gap and the number of people behind each percentage.
The result includes a p-value, which describes how unusual the observed difference, or a more extreme one, would be under that no-difference assumption. A small p-value gives stronger evidence against the assumption. It is not the probability that your conclusion is wrong.
A common rule is to flag results below 0.05, often described as testing at the 95% confidence level. A result that meets that rule is called statistically significant. This means the evidence crosses a chosen threshold; it does not mean the result is certain or necessarily important for the business.
You will also see a confidence interval: a range that helps communicate uncertainty about the size of the difference. A narrow range suggests a more precise estimate. A wide range tells you there is still considerable uncertainty about how large the advantage might be.
You do not need to calculate these figures yourself. The tools do that for you. Understanding what they mean helps you use the answer sensibly.
One question to answer: use the single-comparison tool
Back at the café, your question is specific: does Advert A generate more interest than Advert B?
That is where the Two-Proportion Z-Test Calculator fits.
You enter each percentage and its base, meaning the number of people whose answers produced that percentage. Alternatively, you can enter the number of positive responses and the total number of respondents.
With 12 positive responses out of 20 for A and 10 out of 20 for B, the result does not meet the usual 95% threshold. The p-value is about 0.53, above the usual 0.05 cutoff. A is ahead in your sample, but there is not enough evidence to confidently distinguish the two adverts using this test.
With only 20 people per group, smaller differences can be difficult to distinguish from sampling variation. Small samples also deserve care because the test relies on an approximation that may not be suitable when there are too few positive or negative responses.
That does not prove the adverts perform equally. It means this small study leaves the question unresolved.
Now imagine a larger test. You randomly assign 400 potential customers to two separate groups, and each person sees just one advert. The percentages come out the same:
| Result | Advert A | Advert B |
|---|---|---|
| Would consider visiting | 60% | 50% |
| Positive responses | 120 | 100 |
| Total respondents | 200 | 200 |
This time the difference meets the usual 95% threshold in a two-sided test, which checks for a difference in either direction. The p-value is about 0.044, just below the usual 0.05 cutoff.
The gap is still 10 percentage points, and there is more evidence behind it than before. Now look at what the calculator reports alongside that result.
The confidence interval runs from roughly 0.3 to 19.7 percentage points. The tool also describes the difference as a small effect using a standard statistical scale. That label does not tell you whether it matters for your business. Advert A is ahead, and the true advantage could be almost nothing, or it could be substantial. This test cannot tell the difference between those two possibilities.
So the honest reading is: A is worth taking forward, and this study is not precise enough to say by how much. If the size of the advantage matters for the decision, for example because A costs more to produce, a larger sample could help you estimate the advantage more precisely before committing.
A useful way to report the finding would be:
More people said they would consider visiting after seeing Advert A than Advert B: 60% versus 50%. The results support taking A forward for further testing, although we still need a clearer picture of how large its advantage might be.
Notice that the conclusion stays close to what you measured: stated interest. You have not yet shown that A will bring more people through the door or generate more sales.
The single calculator is useful whenever you have one focused comparison like this. You might compare satisfaction between two separate customer groups, interest in two product concepts, or one survey measure across two waves with different respondents.
See how your result was reached
Scroll to the bottom of the single-test tool and you will find a step-by-step explanation using the numbers you entered.
It walks through your percentages and group sizes, how much the difference could vary from one sample to another, and how the calculation leads to the result. You can follow the reasoning without doing the maths yourself.
This is useful in three ways:
- Check the result. See where the figures come from and compare the calculation with another source if you need to.
- Learn from your own example. Use familiar numbers, such as the café's two adverts, to understand how the test works.
- Explain it to someone else. Use the walkthrough to help answer a colleague or client who asks how you reached your conclusion.
You can use the calculator for a quick check, then explore the explanation whenever you want to understand more.
A whole page of questions: use the multiple-comparison tool
Suppose your café grows, and you now want to understand the experience of new customers compared with regular customers.
You survey both groups about 12 things: menu choice, value, service, cleanliness, ease of ordering and more.
Now your task has changed. You are looking across a table and asking, "Where do these groups differ?"
This introduces another kind of uncertainty. Every test offers an opportunity to flag a difference by chance. As you run more tests, there are more opportunities for a chance finding to appear somewhere in the results.
When you check many measures at once, the chance of flagging at least one difference by accident increases. An adjustment helps account for that extra searching.
The Multiple Two-Proportion Z-Tests Calculator helps account for this. It runs up to 50 comparisons together and offers an adjustment for multiple testing.
That adjustment does not change your survey percentages. It changes how strong the evidence needs to be when you assess a collection of results.
The tool includes a worked example you can load to explore this: 12 brand attributes, two audiences and 500 people in each group. Tested individually, six attributes come back significant. Apply the default adjustment and one survives. The other five are marked "lost to correction" in the results table, with their original p-values still visible beside the adjusted ones.
Back at the café, you might see a similar pattern. Your initial checks might flag several areas. After adjustment, ease of ordering might be the one that holds up while the other gaps no longer meet the threshold.
You now have a clearer starting point for investigation: why do new customers find ordering harder? Perhaps the menu is unfamiliar or the queue is confusing. The test cannot tell you the cause, but it can help you identify a question worth exploring.
The other gaps do not disappear. You simply have less reason to treat them as established differences based on this study alone.
Choosing between the tools
Both tools compare percentages. The main difference is whether you are answering one focused question or interpreting a collection of comparisons together.
| Your situation | Which tool fits? | Why? |
|---|---|---|
| Comparing interest in two adverts on one measure | Single calculator | You have one specific comparison. |
| Comparing one satisfaction measure between two separate groups | Single calculator | You want to examine one gap in detail. |
| Comparing new and regular customers across 12 experience measures | Multiple calculator | You are looking for differences across a set of results. |
| Reviewing several brand attributes between two audiences | Multiple calculator | The adjustment helps account for searching across many comparisons. |
Running each row separately through the single calculator does not remove the multiple-testing issue. The choice depends on how you intend to interpret the findings together.
A few habits that make the results more useful
Start with the decision you want to make. Before looking at the results, decide what you are comparing and what would matter in practice. For the café, a small increase in interest might not justify a much more expensive advert.
Use the correct base. If 200 people completed a survey but only 150 answered a particular question, use the 150 relevant respondents for that question. Enter each group's own base, even when the group sizes differ.
Keep the groups separate. These tools assume independent groups. Comparing the same customers before and after a change requires a method that accounts for those repeated responses. Comparing new customers with an overall total that includes those same customers also breaks the assumption.
Account for weighting when it applies. Some surveys give certain responses more influence to better represent an audience. Unequal weighting can make results less precise than the raw sample size suggests. Ignoring that effect can overstate your confidence. Both tools can account for it. The single calculator accepts an effective sample size or a design effect for each group. The multiple calculator takes one design effect per group and applies it across every comparison. Ask your research provider for the appropriate figure rather than guessing. This adjustment does not fix every possible survey-design issue.
Choose your settings before looking for a winning result. A 95% confidence level with a two-sided test is a common starting point. In the multiple tool, the default Benjamini-Hochberg adjustment aims to limit the expected proportion of false discoveries among the results flagged. Holm-Bonferroni takes a stricter approach to limiting the chance of any false positive across the set. Choose the approach based on the purpose of the analysis, rather than switching settings until a preferred result becomes significant.
Read beyond the significance label. Look at the size of the gap, the uncertainty around it and the practical consequences. A tiny difference can be statistically significant with a large sample. An important-looking gap can remain uncertain with a small one. The advert example above is significant and still leaves the size of the advantage wide open. Pay attention to the calculator's warnings about small bases or unsuitable inputs.
Turning a result into a better decision
At the beginning, the café owner had two percentages and a tempting conclusion: 60% beats 50%, so choose Advert A.
By asking a few more questions, that decision becomes better informed. How many people answered? Were the groups comparable? Was this the only comparison, or one of many? Is the potential advantage large enough to matter?
That is why I built these tools. They give you a practical way to examine the evidence before turning a gap in a table into a story about your customers.
Both are free to use in your browser. Start with the question you want to answer, choose the tool that fits, and use the result alongside what you know about your customers and your business.
