Choosing an Appropriate Quantitative Research Design

When approaching the process of empirical research there are a number of steps we take.

✦ Identify and define research problem and questions. These questions generally come as a broad research question followed by some more targeted sub-questions. A number of studies formulate hypotheses on basis of experience, theory and prior research, although not all research papers state these.

✦ Design research study to collect data bearing on research questions. The subjects for the study and the data you collect will be influenced by access and availability. We aim to conduct the best research study possible, but it is important to consider what is feasible for you. Below, we discuss different types of research designs and later in the blog some guidelines for choosing an appropriate design.

✦ Conduct the research and collect data. It is very important to have a research management plan in place that carefully details how and when you will collect data, who will collect data, how data will be cleaned and stored for analyses.

✦ Analyze the data (through statistical methods).

✦ Interpret the data in light of the research questions.

✦ Formulate conclusions.

Research design refers to how the study was conducted and outlines detailed plans and procedures for how the study will be conducted. Different designs control for different potential threats to the validity of the study. Validity is the extent to which the interpretation of the results of the study follows from the study, in other words the truthfulness or warrant of conclusions. The question of interpretation of results is a critical question for researchers and consumers of research. Whenever we encounter conclusions from a research study we must consider their trustworthiness. Validity is the degree to which scientific explanations of phenomena match reality.

There are many, many different specific research designs, all of which have their own weaknesses. Reading studies by other researchers gives you a sense of the basic designs and some of their challenges. Remember that the goal of research is to use a research design that will result in drawing the most valid and credible conclusions and interpretations. The goal is to minimize alternative plausible explanations for the findings.

For all studies, after we get results, we interpret or draw conclusions from the results. The goal of sound research is to rule out alternative explanations or interpretations of results and to control sources of error, sources that aren’t related to the study’s results. In designing the study, we look to minimize any threats to validity. There are different types of validity we need to consider in our research design, and we look to identify any potential threats to validity and minimize those threats whenever possible.

✦ Statistical Conclusion Validity: the appropriate use of the statistical tests to determine whether purported relationships are a reflection of real or actual relationships.

✦ Construct Validity: the extent to which the measurement of the variables actually represents the targeted underlying construct. Are we measuring what we think we are measuring?

✦ Internal Validity: the extent to which the study establishes a trustable relationship between variables and the extent to which the design of the study minimizes alternative explanations for the findings.

✦ External Validity: the extent of the generalizability of the results and conclusions to other people and locations.

Different designs can counteract or reduce alternative plausible explanations of our findings because they can control for or rule out other explanations. In addition, different designs can better minimize potential threats to validity. There is no such thing as a perfect design, so as researchers we do what we can with the resources we have to design studies in the way that best answers our questions. Let’s take a deeper look at some of the most common designs. Even if your goal is not to conduct quantitative research, familiarity with the designs, and being able to identify and understand the research design being used, will help you become a better consumer of research.

Pre-experimental designs include one-shot case studies, one group pre-test, post-test design, and intact-group comparisons. This type of research design does not have an adequate control group. Subjects are not randomly assigned to groups. This design type does not control for internal validity. These designs provide a quick and convenient method for collecting pilot data that can provide rich information as a pilot study for planning future studies.

True Experimental designs include studies with: a control group design and a post-test only; a control group design and a pre- and post-test; and factorial experimental design. These designs include one or more control groups and use random assignment when placing subjects in a control or treatment group. These are ideal for assessing causality, and almost all threats to internal validity are ruled out.

Quasi-experimental designs include non-equivalent control group designs, where the assignment of participants to groups is not controlled by the investigator. This design is used when the researcher is unable to randomly assign subjects to treatment groups owing to moral, ethical or pragmatic reasons. When experimental design is not feasible, a quasi-experimental design is the next best design for reaching causal interpretations. However, there is a need to be careful. Causal interpretations from these designs are more tenuous than causal interpretation from experimental designs, but more tenable than interpretation from pre-experimental designs. Quasi-experimental designs also include time series studies.

Ex Post Facto designs include correlational designs and criterion group designs, where two groups of subjects are compared on one measure. Ex post facto studies look at the degree of association between two or more variables; they do not examine causal relationships. In these studies, treatment was already administered by nature (differences in environments, inheritances or combo). In other words, the researcher doesn’t do anything to sample, but just studies people for whom something has happened. For these designs, causal interpretations are not warranted.

The conclusions we draw (relationships, causality) and inferences we make from our statistics (generalizations to larger populations) are only as good as the research design we start out with. So, our goal is to build research designs that will lead to the most valid and credible conclusions. All along the way of planning one’s research, important decisions need to be made by a researcher. These decisions will be influenced by a number of factors including availability of subjects, time, resources, scope of work, and the specific expertise of the researcher. To select the best research design for your study, you start by narrowing down your research question(s). Next, there are a number of things for you to consider.

1. Are you looking to conduct a pilot study to generate some directions for future research? In which case, you could use a pre-experimental design.

2. Is your goal to establish a relationship between two variables where you are not administering a treatment? If so, this would lead to an Ex Post Facto design.

3. Are you seeking to establish causality? In which case, you would use an experimental design.

4. Can you randomly assign to treatment and control groups? Then you can use a true experimental design.

5. Is assignment to groups not within your control? In which case, you would use a quasi-experimental design.

How will you identify and minimize potential threats to validity? Charlesworth Author Services offers offering statistical analysis for researchers in the field of medical and life sciences articles. This service will help the researcher improve the accuracy and reporting of their data prior to submitting their article to a publisher.

To find out more about this service please visit: How does the Statistical Review Service Work?

Join us on our FREE series of webinars designed to help you understand statistics and data presentation for publication.

Maximise your publication success with Charlesworth Author Services.