I carried out a questionnaire exercise 4 years ago, attempting to replicate (as far as possible with only me as the resource) a questionnaire study done over a period of 3 years, with an initial population of 128,000, reducing to 2,000 by the end of the study by issuing a single online questionnaire to a very much smaller population based in a different part of the same country, whose population had not been part of the initial exercise. The results are all in SPSS but I also have them available in Excel. I carried out a descriptive analysis of the findings from my questionnaire and compared my results with the same questions from the original survey but am now being asked to compare my results in more depth with those from the original questionnaire. My statistical knowledge is sketchy 🙁 so I am not even sure whether the question I am going to ask here is a “good one”!). For both studies, only gender and age bands are known and I was wondering whether it would be possible to use the two sample t test to say whether the two populations are similar (or not) in their responses to individual questions in the two questionnaires. I have several books that I have consulted and online papers and I still can’t answer the question. Are you able to shed any light please? Thank you.

This calculator uses the following formulas to compute sample size and power, respectively:
$$
n_A=\kappa n_B \;\text{ and }\;
n_B=\left(\frac{p_A(1-p_A)}{\kappa}+p_B(1-p_B)\right)
\left(\frac{z_{1-\alpha}+z_{1-\beta/2}}{|p_A-p_B|-\delta}\right)^2$$

$$1-\beta=
2\left[\Phi\left(z-z_{1-\alpha}\right)+\Phi\left(-z-z_{1-\alpha}\right)\right]-1
\quad ,\quad z=\frac{|p_A-p_B|-\delta}{\sqrt{\frac{p_A(1-p_A)}{n_A}+\frac{p_B(1-p_B)}{n_B}}}$$
where

- $\kappa=n_A/n_B$ is the matching ratio
- $\Phi$ is the standard Normal distribution function
- $\Phi^{-1}$ is the standard Normal quantile function
- $\alpha$ is Type I error
- $\beta$ is Type II error, meaning $1-\beta$ is power
- $\delta$ is the testing margin