Any quantitative academic study or experiment, which involves a comparison between two or more variables, requires a means of determining how substantially different these variables are from one another. The way in which disparities between variables either in terms of their differences or similarities can be quantified and is referred to as the effect size. This article will delve into statistical and practical significance, as well as how to calculate effect size.
Definition: Effect size
To give a concise definition, the effect size, quite simply, is the size (or magnitude) of an effect. A large effect size indicates that a considerable degree of significance can be attributed to the data in question, while a small effect size suggests that the veracity of research results will be negligible.
Note: report results can be presented in one of several styles, however, this article follows the APA style guidelines.
Effect size: Significance
While an equivalence is often assumed with the more general concept of significance, in reality, the effect size is distinct for a number of important reasons. In order to explore this further, it is useful to look at what role significance, and ‘statistical’ significance, in particular, plays in the effect size.
Statistical significance shows that an effect exists in a study. Statistical significance is represented by calculating and assigning a p-value, or probability value, to data. Using the concept of the null hypothesis whereby an inconclusive, random result is considered to be devoid of statistical relevance (ergo “null”) as a starting point, a low p-value indicates that the reverse is true, i.e. an acceptable degree of statistical relevance is in evidence.
However, statistical significance can be misleading as it does not take into account the sample size. Increased sample size will achieve a closer resemblance to “real-world” conditions, moving researchers closer to establishing a causal relationship between two factors.
Research samples, which are large enough to achieve an approximation of “real-world” conditions, as mentioned above, are considered to have practical significance.
While not dependent on sample size, research indicates that the variability of effect sizes is diminished with increasing sample size.
It is important to report the effect size in research papers in order to indicate the practical significance of any data, which results from a given research project. APA guidelines require published research to include effect sizes and confidence intervals (a method of describing the uncertainty inherent in an estimate) whenever possible.
Statistical significance vs. practical significance: Example
A wide acknowledged exampe of and exemplary experiment is “Visual Adaption of the Perception of Causality” (Rolf et al, 2013).
Effect size: Calculation
While there are many different measures to calculate the effect size, the two which are used mostly are Cohen’s d and Pearson’s r.
In simple terms, Cohen’s d measures the difference in size between two groups, while Pearson’s r measures the strength of the relationship between two variables.
Calculating the effect size with Cohen’s d
Designed in order to provide researchers with a clear method by which to compare two groups, Cohen’s d measures the effect size as a number of standard deviations.
This is accomplished by subtracting the mean value of group two from the mean value of group one (M1 – M2) and dividing the result by the pooled standard deviation (SD). This may be expressed as an equation as follows:
d= (M1 – M2) / SD pooled
A result is a single number that summarizes the variability in a dataset, whereby a higher number indicates that more data points are further away from the mean. In other words, a dataset that has been assigned a higher number according to this method, will exhibit more frequent occurrences of extreme discrepancies between individual values within the data.
The key elements of Cohen’s d method may be summarized as follows:
|Pooled standard deviation||The average degree to which individual values differ within a given dataset.|
|Standard deviation from a critical group||The average degree to which individual values differ within a subset of a given dataset deemed to be of particular significance.|
|Standard deviation from the pretest data||The average degree to which individual values differ within a dataset collected by researchers as a preparatory measure prior to commencing collection of their main dataset.|
Calculating the effect size with Pearson’s r
Pearson Correlation or Pearson’s r measures the effect size as an extent of a linear relationship between two variables. This measurement will indicate whether two factors move in the same direction (a negative correlation), or in the opposite direction (a positive correlation).
This spectrum of positive and negative correlations establishes a range between +1 and -1 within which Pearson’s r measures are quantified. A Pearson’s r measurement of 0 (a neutral rating) between two factors indicates that the factors in question do not have an effect on one another.
Calculating a Pearson’s r value requires the use of statistical software in order to generate visual graphs for use in the interpretation of the dataset presented. Pearson’s r may be presented as a formula as follows:
The various elements included in the formula are as follows:
- Correlation Coefficient
- Value of the x-variable included in a sample
- Mean of the values of the x-variable
- Value of the y-variable in a sample
- Mean of the y-variable values
As discussed above, effect size is the size or magnitude of the relationship between two variables. This relationship is presented as a numeric value.
In order to calculate and assign a numeric effect size value to data, researchers take the differences between a pair of groups and divide it by the standard deviation of one group in the pair.
Effect size is significant because it equips readers with a quantifiable numeric value by which the relevance of data can be reported and assessed.