Research Methods & Design (Practice)

The Research Design

  • When designing a research study, we try to establish two or more types/levels of one variable (independent variable) in order to observe its effects on other variables (dependent variable).
  • Research Methodology: The procedures and activities used to collect and analyze a set of empirical data bearing on some questions of interest.
  • Research Design: The master plan or blue-print for conducting a study whose results can be interpreted with minimal ambiguity
  • In all research, the goal is to show that the variability of the dependent variable is due to the variability of the independent variable, and not due to other factors.
  • When establishing a research design, consider the following questions:
  1. Who or what will be the focus of the study?
  2. What attributes of individuals, groups or organizations will be observed?
  3. In what contexts will observations be made?
  4. What methods will be used to analyze the data produces?
  5. How are we going to deal with relevant and nuisance variables?
  • The research strategy/final choice of a study depends on the following factors:
    • Patterns established by previous research
    • The constraints and resources available
    • The needs or preferences of the sponsors or the ultimate consumers of the research
    • The personal strengths, capacities, and expertise of the researcher

Human Resource Research Design

  • Research in HR usually takes form in the following 4 research designs:
  1. Surveys
  2. Experimental
  3. Quantitative
  4. Existing research and meta-analysis

Survey Research

  • Can be used to gauge people’s opinions or attitudes about a certain topic
  • Used to examine the relationship between variables
  • Mostly used to test the relationships between:
    • Individual perceptions
    • Individual ability
    • Individual personality; and
    • Work outcomes
  • Problems: Surveys can only provide the most indirect evidence for cause-and-effect relations.

Experimental Research

  • This type of research provides the strongest evident for cause-and-effect relationships
  • Plays an important role in developing and assessing organizational policies and practices
  • Allows you to form conclusions with the least ambiguity
  • Most appropriate to use when trying to learn about the effect of an intervention of program on some outcome
  • Problems: Experimental research has theoretical and practical limitations

Qualitative Research

  • Qualitative Research: The gathering and sorting of information through informal methods
  • For qualitative research, managers or researchers may rely on the personal experiences and comments of colleagues
  • The method is most useful when trying to obtain a rich, in-depth description of some event or activity.
  • In HR, qualitative methods are often used I combination with other research methods.
    • Example: Job analysis interviews after often used along with survey data to determine the tasks and responsibilities required to perform the job

Existing Research and Meta-Analysis

  • Instead of conducting new and unnecessary research, useful information can be obtained by reviewing and summarizing existing research
  • Meta-Analysis: Technique that allows researchers to combine the results of existing research in a particular area and to calculate the overall effect on an intervention or the correlation between a predictor and outcome(s).
  • This method helps with:
    • Developing theory
    • Identifying research needs in a particular area
    • Determining the strength of a relationship between variables
  • This method is useful for small organizations that don’t have the ability to conduct large-scale research studies

Choosing a Research Design

  • Failure to choose an appropriate research design can prevent the researcher from:
    • Answering important research questions; and
    • Can restrict the interpretation of research findings
  • A research design must meet several criteria:
  1. First, it must allow the research to adequately test the hypothesis and answer the questions they’re asking
    • If a research design won’t allow the researcher to properly test the hypothesis, then it shouldn’t be considered
  2. Second, nuisance or extraneous factors that could influence the results of a study must be identified and controlled
    • Failure to do so can severely limit the researcher’s ability to make sound conclusions about the relation between an independent and dependent variable
  3. Third, the results must tell us how other people would react in different contexts
    • Failure to do so can result in inaccurate conclusions
  4. Finally, the research design should be efficient—we want to get the information we need with the available resources

Test Study Hypothesis                                                      

  • Test to see if the hypothesis is true
  • The hypothesis should dictate who the participants should be (e.g. nurses, managers, students, etc.) and what methods should be used to analyze the data.
    • Example: If you’re trying to see the effect of a program on some outcome variable(s), you’d more likely choose experimental design instead of survey design.

Control Nuisance and Extraneous Factors

  • Nuisance and Extraneous Factors: Variables that the research isn’t usually interest in studying, but might interfere with or influence the relation between an independent variable and a dependent variable.
    • Example: In a study of the relationship between age and job performance, a researcher might find that age is positively related to job performance. However, if older workers have more experience then their job performance could be more related to their experience than their age.
    • If the research did not consider those other variables, their research conclusions could be misleading
  • It’s important to identify all nuisance factors and then choosing a design that allows you to control them.
  • Failure to control nuisance variables in a study (especially in an experiment) is often considered to be a fatal flaw and is likely to result in the rejection of any inferences, conclusions, or assertions that may be based on the findings.
  • Nuisance and extraneous factors can be controlled by simply measuring them and including them in the data analysis as statistical controls or through the use of an experimental design that has high internal validity

External Validity

  • External Validity: Sampling randomly from a defined population can help you use the data from that sample to make statements about the whole population with specified levels of confidence.
    • Also known as generalizability as it reflects the extent to which the results of a study will hold up or generalize to other samples of participants and to other contexts
  • Limitations: The nature of the sample used and the setting could influence the results and limit the generalizability of the results to other samples and settings
  • Experiments have an issue of external validity because they’re often done in laboratory studies which take place in an artificial setting and often involve student participants to organizational settings. A.K.A. it’s not a good representation of the real world.
  • Surveys have an issue of external validity because if the study uses participants who aren’t randomly sampled, the survey may not generalize to other groups of participants.
    • This is a problem because the ability to generalize the results of a study to a larger population and to other organizations is often an important consideration when choosing a research design

Statistical Conclusion Validity

  • Statistical Conclusion Validity is less concerned with bias than with sources of error variance, considering that anything which might contribute to randomness in the data would be seen as a threat.
  • Factors that are relevant to statistical conclusion validity include the reliability of measures and sample size
    • Example: if a variable has low reliability or a study involves a small number of participants, there is an increased likelihood that the researcher won’t detect real differences between experimental groups or real relationships between variables even when they do exist. As a result, it’s possible to reject the hypothesis although it’s actually correct.
  • Null Hypothesis: Hypothesis that says there’s no significant difference between specified populations
  • If a research rejects the null hypothesis and concludes that there’s a relationship between variables when there really isn’t any, then the research has made a type I error.
  • Type I Error: The probability that we will conclude that there’s a significant effect or relationship even when none exists.
  • Type II Error: The probability that we will conclude that there is no effect or relationship between the variables even through one exists.
  • To avoid these errors, it’s important that the research design allows the research to make valid statistical conclusions.
  • Sample sizes must be large enough to detect type I and type II errors.

Research Design Efficiency

  • Resources needed for research design efficiency include:
    • Time available for research
    • The number of participants (cases) that can be studied
    • The funds that exist for staff time, instruments, inducements, etc.
    • The level of expertise that the researcher has in using various designs

Research Design Issues

The Research Context

  • HR research can be conducted in a number of research contexts. Most often it’s desirable to conduct the research in an actual organization.
    • This might involve one organization, such as an experiment to test the effects of a new training program.
    • In other cases, the research might involve several organizations to allow the researcher to study differences in particular human resource programs or policies.
  • Human resource research can also take place in laboratory settings and involve individuals who aren’t currently employed
    • Example: Research on realistic job previews, which involves presenting job applicants with both positive and negative features of a job and organization, has been conducted with samples of actual job applicants and employees in an organizational setting, as well as with students who participated in a contrived laboratory setting.

The Nature and Size of the Sample

  • In most cases, research is done with a sample of individuals instead of a population
  • Two features of the sample that are particularly important are the sample characteristics and sample size
  • Sample Characteristics: The nature of the sample relative to the population of interest
  • Sample size will affect representativeness. It is harder for a small sample to be representative in comparison to a population size.



Data Sources

  • The research design implies who (or what) is going to be measured, assessed, tested, or monitored.
  • It’s common to use self-reports in HR research, where individuals who are the focus of the study (e.g. employees) also provide the data (e.g. individual characteristics, job attitudes, job performance, etc.)
  • Self-reports are used in the following ways:
  1. Obtaining demographic or otherwise factual data (e.g. years of work experience)
  2. Assessing the effectiveness of instructions or experimental manipulations
  3. Gathering personality data
  4. Obtaining descriptions of a respondent’s past of characteristic behavior, or how they would behave under hypothetical conditions
  5. Measuring internal or psychological states of respondents (e.g., work attitudes)
  6. Assessing respondents’ perceptions of context or organizational variables
  • Problems: Issues with self-reports include trying to make a correlation between two or more variables that are collected from the same respondent. This relates to the potential problem of common method variance.
  • Common Method Variance: When the measures that come from the same source are spuriously (falsely) related
  • Self-reports also have the potential to induce socially desirable responses—responses that are intended to make the respondent look good in the eyes of the investigator or society at large.
    • Example: Asking someone in a survey of interview if they “learned anything” as a result of a training program is likely to produce an affirmative answer
  • Ways to deal with some of the problems of self-report:
  1. Using a research method that involves separation of measures
    • This might require obtaining self-report data at slightly different times or using different designs
  2. Ensuring that some variables or some of the data are obtained from more than one source (e.g., the employee and their peers or supervisors).
  • It’s common to take a test before and after a training program to see the effectiveness of the training, however it is said that any comparison of pretest and pro-test scores from self-report measures may be misleading because of response shift bias.
  • Response Shift Bias: A change in the internal standards used by a respondent in making a judgement about his or her level of knowledge or functioning
    • Basically, after a person has gone through training, they’re likely to reconsider any self-reports or self-assessments. They might think they’re not as skilled as they thought they were considering the competency of the instructor, the nature of the course content, the behavior of their classmates, etc.
  • Researchers must remain alert when several measures are taken over time to assess a change in employees’ perceptions, attitudes, or behaviors.
  • Not all changes as “actual” changes in the variable of interest. There are 3 possibilities known alpha, beta, and gamma changes—the reasons for changes in an individual’s responses to a given survey item or variable over time.
  • Alpha Change: Reflecting actual changes in the level of the item or variable
    • Example: A moderate rating of job satisfaction at Time 1 (a rating of 3 on a 5-point scale) becomes stronger at Time 2 (a rating of 4).
  • Beta Change: A recalibration of the levels
    • Example: The “3” at Time 1 is interpreted as a “4” at Time 2
  • Gamma Change: A redefinition of the construct or variable itself
    • Example: The very meaning of “jo satisfaction” is transformed