Conducting Research Projects Guide: Statistical tests

Research uses variables (things that can be measured) to prove or disprove an hypothesis.  Variables can be categorical (e.g. gender) or numerical (e.g. 1.5).  

All research is based on a question the researcher wants to test from which an hypothesis can be developed (e.g. More people like milk chocolate than dark chocolate).  To test this hypothesis,  the following process can be used:

Definitions:

Hypothesis (H1):  the proposed idea based on limited data that is used as the starting point of an investigation or experiment.

Null hypothesis (H0): assumes there is no difference between the variables being studied. Statistical testing is used to prove or disprove the null hypothesis allowing the researcher to draw conclusions.

P-value: the probability or likelihood of a certain result when applied against a statistical test.  This should be determined at the start of the test and is usually set at 0.05

Alpha value:  this is the cut off point of the P-value and refers to the probability of the result being due to chance.  The smaller the alpha value, the more unusual the result.  It is usually set at 0.05.  The alpha value can also calculate the confidence level.

Confidence level:  how confident the researcher can be that certain results can be expected.

Parametric tests:  make a number of assumptions about the population and the nature of the sample. Their purpose is to test hypotheses.

Non-parametric tests:  do not have strict assumptions and are more suited to small samples or when data are ranked.

(Pallant, 2013, p.212).

The information in this box is based on chapters from Pallant, 2013.  This is an excellent resource for understanding statistical testing.

Parametric tests have 5 main assumptions:

Level of measurement: Parametric tests assume the dependent variable is measured using a continuous scale.  This allows for greater choice in analysis techniques.

Random sampling:  It is assumed that the sample is a random group chosen from the population.

Independence of observations:  Parametric tests assume that each observation or measurement is independent of each other and are not influenced in any way by each other.

Normal distribution:  It is assumed that the population as a whole is normally distributed (i.e. it fits the bell curve).

Homogeneity of variance: Parametric tests assume that the variability of scores for each group in the study is similar.

Parametric tests can have two error types:

1. The null hypothesis is rejected despite being true. This can be minimised by selecting the appropriate alpha level.

2. Failing to reject the null hypotheses when it is false.  

Following is a sample of the most common parametric tests:

T-Tests:

• determines whether the differences between the means of two variables is statistically significant.  
• two kinds of t Tests: independent (the variables are independent of each other) and paired (the two variables are connected in some way).
• the calculation involves the mean values of each group, the standard deviation and number of data values.
• for the paired sample T-test, is used to test changes in scores e.g. at Time 1 and then later, after an intervention or event, at Time 2. The same participants are tested each time.
• independent T-tests are used to compare scores between two different groups. In this case, data is only collected on one occasion but uses two different sets of participants.
• data is from randomly selected portions of the population.
• the distribution of the results will conform to the bell curve.
• when the test accepts the null hypothesis, the differences are said to be statistically significant.  If the test disproves the null hypothesis, the differences are said to be not statistically significant.

ANOVA Tests

Analysis of Variance tests (ANOVA) are used when there are two or more groups or time periods. it is used to test whether there are main effects for each variable and whether any interaction between variables is statistically significant. There are four types of ANOVA tests:

• One-way between groups - used when there is only one independent variable (e.g. gender). This test has one independent categorical variable with three or more categories and one dependent continuous variable.  It tests the difference between the mean scores on the dependent variable across the three groups.
• One-way repeated measures - each participant is exposed to two or more different conditions or occasions. This test has one independent categorical variable and one dependent continuous variable.  It tests  whether or not there is a difference among the three sets of data that is statistically significant.
• Two-way analysis of variance between groups - used when there are two independent variables (e.g. gender and age). Uses two categorical independent variables and one continuous dependent variable.  This test is used to determine the effect of each independent variable on the dependent variable and any interaction.
• Mixed between-within groups - has one independent variable between the groups, one independent variable within groups and one continuous dependent variable. Uses one categorical independent variable between groups, one categorical independent variable within groups and one continuous dependent variable.  This test is used to determine the effects of each independent variable and any interaction between the two.

MANOVA Test

The multivariate analysis of variance test is an extension of the ANOVA test and is used when there is more than one dependent variable that are related in some way. This tests the possibility that the difference between groups when the variables are combined is due to chance. MANOVA can use one-way, two-way or higher-order designs (multiple independent variables) or when analysing covariance using one extra variable as the control. 

The MANOVA tests the null hypothesis in terms of whether the means on a set of dependent variables do not vary across the groups.

ANCOVA

Analysis of Covariance allows you to test whether an additional continuous variable has some influence on the scores of the dependent variable.  This test can be used as part of the one-way, two-way and multivariate ANOVA tests.  It is particularly useful when comparing the impact of two different interventions on the groups when using pre-test and post-test design. When the sample size is small and the effects are small to medium, ANCOVA can be a good choice.  It does require you to carefully choose the covariates that will be used as controls.  They should be continuous, measured reliably and the correlation with the dependent variable should be significant.

(Pallant, 2013).

The information in this box is based on chapters from Pallant, 2013.  This is an excellent resource for understanding statistical testing.

Non-parametric tests are best for small sample sizes or when the data is ranked. 

They have less strict assumptions.

The following are types of non-parametric tests:

Chi-square for goodness of fit:  Used to compare the proportion of cases with hypothesised values or those from a previous comparison population.  This test involves one categorical variable with two or more categories e.g. drinker (Yes/No) and a hypothesised proportion (e.g. 20%).   It tests whether or not there is a significant difference between the variables.

Chi-square for independence: This test is used to determine the relationship between two categorical variables, each of which can have two or more categories. It compares the proportion of cases in each of the categories, measuring the values that would be expected if there was no association between the variables.  It results in a crosstabulation table.

McNamar's Test:  This test is used when you have matched or repeated measures design (e.g. pre- and post-test).  It uses two categorical variables  measuring the same characteristic tested at two time frames (pre intervention and post-intervention). Both categories must be tested at both time frames.

Cochran's Q Test: Cochran's Q Test is used when there are three or more time frames. it uses three categorical variables measuring the same characteristic collected from each participant multiple times.

Kappa Measure of Agreement: This is mostly used in medical research to assess the agreement between two different clinicians (raters) or consistency of two different diagnostic tests.  It estimates the proportion of agreement between the two raters taking into account the amount of agreement that could be reached by chance.

Mann-Whitney U Test: This test measures the difference between two independent groups on a continuous measure. Unlike the T-test, the Mann-Whitney U Test compares the median. It uses ones categorical variable with two groups and one continuous variable.

Wilcoxon Signed Rank Test: This test is used when there are repeated measures. e.g. When the participants are measured in two different time-frames or two different conditions.  The results are compared.

Kruskal-Wallis Test: This test allows the researcher to compare scores on a continuous variable for three or more groups. The scores are ranked then the mean rank is compared.

Friedman Test: Measures the same sample of participants at three or more time-frames or three different conditions.

The above video clip is an excellent introduction to statistics.  It is easy to follow with excellent examples.

The above video outlines seven different statistical tests, then uses 7 examples to show how to choose which test to use.