The application of factorial surveys to study recruiters’ hiring intentions: comparing designs based on hypothetical and real vacancies

Gutfleisch, Tamara Rebecca; Samuel, Robin; Sacchi, Stefan

doi:10.1007/s11135-020-01012-7

Article (Scientific journals)

The application of factorial surveys to study recruiters’ hiring intentions: comparing designs based on hypothetical and real vacancies

Gutfleisch, Tamara Rebecca; Samuel, Robin; Sacchi, Stefan

2020 • In Quality and Quantity

Peer Reviewed verified by ORBi

Permalink
https://hdl.handle.net/10993/44075

DOI
10.1007/s11135-020-01012-7

Files (1)Send to Details Statistics Bibliography Similar publications

Files

Full Text

Gutfleisch2020_Article_TheApplicationOfFactorialSurve.pdf

Publisher postprint (768.63 kB)

Download

All documents in ORBilu are protected by a user license.

Send to

RIS BibTex APA Chicago Permalink X Linkedin

Details

Keywords :

Factorial survey experiments; Hiring intentions; Real vacancies; Hypothetical vacancies

Abstract :

[en] Factorial survey experiments have been widely used to study recruiters’ hiring intentions. Respondents are asked to evaluate hypothetical applicant descriptions, which are experimentally manipulated, for hypothetical job descriptions. However, this methodology has been criticized for putting respondents in hypothetical situations that often only partially correspond to real-life hiring situations. It has been proposed that this criticism can be overcome by sampling real-world vacancies and the recruiters responsible for filling them. In such an approach, only the applicants’ descriptions are hypothetical; respondents are asked about a real hiring problem, which might increase internal and external validity. In this study, we test whether using real vacancies triggers more valid judgments compared to designs based on hypothetical vacancies. The growing number of factorial survey experiments conducted in employer studies makes addressing this question relevant, both for methodological and practical reasons. However, despite the potential implications for the validity of data, it has been neglected so far.We conducted a factorial survey experiment in Luxembourg, in which respondents evaluated hypothetical applicants referring either to a currently vacant position in their company or to a hypothetical job. Overall, we found little evidence for differences in responses by the design of the survey experiment. However, the use of real vacancies might prove beneficial depending on the research interest.We hope that our comparison of designs using real and hypothetical vacancies contributes to the emerging methodological inquiry on the possibilities and limits of using factorial survey experiments in research on hiring.

Disciplines :

Sociology & social sciences

Author, co-author :

Gutfleisch, Tamara Rebecca ; University of Luxembourg > Faculty of Language and Literature, Humanities, Arts and Education (FLSHASE) > Integrative Research Unit: Social and Individual Development (INSIDE)

Samuel, Robin ; University of Luxembourg > Faculty of Language and Literature, Humanities, Arts and Education (FLSHASE) > Integrative Research Unit: Social and Individual Development (INSIDE)

Sacchi, Stefan

External co-authors :

yes

Language :

English

Title :

The application of factorial surveys to study recruiters’ hiring intentions: comparing designs based on hypothetical and real vacancies

Publication date :

04 August 2020

Journal title :

Quality and Quantity

ISSN :

1573-7845

Publisher :

Kluwer Academic Publishers, Netherlands

Peer reviewed :

Peer Reviewed verified by ORBi

Additional URL :

https://doi.org/10.1007/s11135-020-01012-7

Funders :

The study was funded by an Internal Research Project Grant of the University of Luxembourg [IRP17 - EDYPOLU].

Available on ORBilu :

since 12 August 2020

Statistics

Number of views

175 (33 by Unilu)

Number of downloads

74 (11 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

Bibliography

Ajzen, I., Brown, T.C., Carvajal, F.: Explaining the discrepancy between intentions and actions: the case of hypothetical bias in contingent valuation. Personal Soc. Psychol. Bull. 30(9), 1108–1121 (2004)
Atzmüller, C., Steiner, P.M.: Experimental vignette studies in survey research. Methodol. Eur. J. Res. Methods Behav. Soc. Sci. 6(3), 128–138 (2010)
Auer, D., Bonoli, G., Fossati, F., Liechti, F.: The matching hierarchies model: evidence from a survey experiment on employers’ hiring intent of immigrant applicants. Int. Migr. Rev. 53(1), 90–121 (2019)
Auspurg, K., Hinz, T.: Factorial Survey Experiments. Sage Publications, Los Angeles (2015)
Auspurg, K., Jäckle, A.: First equals most important? order effects in vignette-based measurement. Sociol. Methods Res. 46(3), 490–539 (2017)
Auspurg, K., Hinz, T., Liebig, S., Sauer, C.: The Factorial Survey as a Method for Measuring Sensitive Issues. In: Engel, U., Jann, B., Lynn, P., Scherpenzeel, A., Sturgis, P. (eds.) Improving Survey Methods: Lessons from Recent Research, pp. 137–149. Routledge, New York (2015)
Auspurg, K., Hinz, T., Walzenbach, S.: Are factorial survey experiments prone to survey mode effects? In: Lavrakas PJ, Traugott MW, Kennedy C, Holbrook AL, de Leeuw ED, West BT (eds) Experimental Methods in Survey Research: Techniques that Combine Random Sampling with Random Assignment. Wiley, Hoboken, pp. 371–392 (2019)
Baert, S., (2018) Hiring discrimination: an overview of (Almost) all correspondence experiments since. In: Gaddis, S.M. (ed.) Audit Studies: Behind the Scenes with Theory, Method, and Nuance, method, ser edn, pp. 63–77. Springer, New York (2005)
Baert, S., As, De Pauw: Is ethnic discrimination due to distaste or statistics ? Econ. Lett. 125(2), 270–273 (2014)
Baert, S., Cockx, B., Gheyle, N., Vandamme, C.: Is there less discrimination in occupations where recruitment is difficult? Ind. Labor Relations Rev. 68(3), 467–500 (2015)
Bertogg, A., Imdorf, C., Hyggen, C., Parsanoglou, D., Stoilova, R.: Gender discrimination in the hiring for skilled professionals in two male-dominated occupational fields–a factorial survey experiment with real-world vacancies and recruiters in four European countries. Kolner. Z. Soz. Sozpsychol. (2020). https://doi.org/10.1007/s11577-020-00671-6
Biesma, R.G., Pavlova, M., van Merode, G.G., Groot, W.: Using conjoint analysis to estimate employers preferences for key competencies of master level Dutch graduates entering the public health field. Econ. Educ. Rev. 26(3), 375–386 (2007)
Bills, D.B., Stasio, V.D., Gërxhani, K.: The demand side of hiring: employers in the labor market. Ann. Rev. Sociol. 43, 6.1–6.20 (2017)
Birkelund, G.E., Janz, A., Larsen, E.N.: Do males experience hiring discrimination in female-dominated occupations? An overview of field experiments since 1996. GEMM working paper, https://gemm2020.eu/wp-content/uploads/2018/12/Gender-Discrimination-a-summary-1.pdf (2019)
Booth, A., Leigh, A.: Do employers discriminate by gender? a field experiment in female-dominated occupations. Econ. Lett. 107, 236–238 (2010)
Brewer, M.B., Crano, W.D.: Research Design and Issues of Validty. In: Reis, H.T., Judd, C.M. (eds.) Handbook of research methods in social and personality psychology, 2nd edn, pp. 11–26. Cambridge University Press, New York (2014)
Carlsson, M.: Does hiring discrimination cause gender segregation in the Swedish labor market? Fem. Econ. 17(3), 71–102 (2011)
Damelang, A., Abraham, M.: You can take some of it with you! a vignette study on the acceptance of foreign vocational certificates and ethnic inequality in the German labor market. Z. Soziol. 45(2), 91–106 (2016)
Damelang, A., Abraham, M., Ebensperger, S., Stumpf, F.: The hiring prospects of foreign-educated immigrants: a factorial survey among German employers. Work Employ. Soc. 33, 1–20 (2019)
Damelang, A., Ebensperger, S., Stumpf, F.: Foreign credential recognition and immigrants’ chances of being hired for skilled jobs—evidence from a survey experiment among employers. Soc. Forces (2020). 10.1093/sf/soz154 DOI: 10.1093/sf/soz154
De Wolf, I., Van Der Velden, R.: Selection processes of three types of academic jobs. An experiment among Dutch employers of social sciences graduates. Eur. Sociol. Rev. 17(3), 317–330 (2001)
Di Stasio, V.: Education as a signal of trainability: results from a vignette study with Italian employers. Eur. Sociol. Rev. 30(6), 796–809 (2014)
Di Stasio, V.: Who is ahead in the labor queue? institutions’ and employers’ perspective on overeducation, undereducation, and horizontal mismatches. Sociol. Educ. 90(2), 109–126 (2017)
Di Stasio, V., Gërxhani, K.: Employers’ social contacts and their hiring behavior in a factorial survey. Soc. Sci. Res. 53, 93–107 (2015)
Di Stasio, V., Van De Werfhorst, H.G.: Why does education matter to employers in different institutional contexts? a vignette study in England and the Netherlands. Soc. Forces 95(1), 77–106 (2016)
Dülmer, H.: Experimental plans in factorial surveys: random or quota design? Sociol. Methods Res. 35(3), 382–409 (2007)
Eriksson, S., Rooth, D.O.: Do employers use unemployment as a sorting criterion when hiring? evidence from a field experiment. Am. Econ. Rev. 104(3), 1014–1039 (2014)
Fernandez, R.M., Mors, M.L.: Competing for jobs: labor queues and gender sorting in the hiring process. Soc. Sci. Res. 37(4), 1061–1080 (2008)
Hainmueller, J., Xu, Y.: Ebalance: a stata package for entropy balancing. J. Stat. Softw. 54(7), 1–18 (2013)
Hainmueller, J., Hangartner, D., Yamamoto, T.: Validating vignette and conjoint survey experiments against real-world behavior. Proc. Natl. Acad. Sci. 112(8), 2395–2400 (2015)
Hox, J.J.: Multilevel analysis: techniques and applications, 2nd edn. Routledge, New York (2010)
Humburg, M., Van Der Velden, R.: Skills and the graduate recruitment process: evidence from two discrete choice experiments. Econ. Educ. Rev. 49, 24–41 (2015)
Imdorf, C., Shi, L.P., Sacchi, S., Samuel, R., Hyggen, C., Stoilova, R., Yordanova, G., Boyadjieva, P., Ilieva-Trichkova, P., Parsanoglou, D., Yfanti, A.: Scars of early job insecurity across Europe: insights from a multi-country employer study, risk factors and policies. In: Hvinden, B., Hyggen, C., Schoyen, M.A., Sirovátka, T. (eds.) Youth Unemployment and Job Insecurity in Europe, pp. 93–116. Edward Elgar Publishing, Cheltenham (2019)
Karpinska, K., Henkens, K., Schippers, J.: The recruitment of early retirees: a vignette study of the factors that affect managers’ decisions. Ageing Soc. 31(4), 570–589 (2011)
Keuschnigg, M., Wolbring, T.: The use of field experiments to study mechanisms of discrimination. Anal. Krit. 38(1), 179–202 (2016)
Koch, A.J., D’Mello, S.D., Sackett, P.R.: A meta-analysis of gender stereotypes and bias in experimental simulations of employment decision making. J. Appl. Psychol. 100(1), 128–161 (2015)
Kroft, K., Lange, F., Notowidigdo, M.J.: Duration dependence and labor market conditions: evidence from a field experiment. Q. J. Econ. 128, 1123–1167 (2013)
Krosnick, J.A., Judd, C.M., Wittenbrink, B.: The measurement of attitudes. In: Albarracín, D., Johnson, B.T., Zanna, M.P. (eds.) The Handbook of Attitudes, pp. 21–78. Psychology Press, New York (2014)
Kübler, D., Schmid, J., Stüber, R.: Gender discrimination in hiring across occupations: a nationally-representative vignette study. Labour Econ. 55, 215–229 (2018)
Kuhfeld, W.F.: Experimental design: efficiency, coding, and choice designs. Marketing Research Methods in SAS, pp. 53–241. SAS Institute, Cary (2010)
Liechti, F., Fossati, F., Bonoli, G., Auer, D.: The signalling value of labour market programmes. Eur. Sociol. Rev. 33(2), 257–274 (2017)
McDonald, P.: How factorial surveys analysis improves our understanding of employer preferences. Schweiz. Z. Soziol. 45(2), 237–260 (2019)
Mergener, A., Maier, T.: Immigrants’ chances of being hired at times of skill shortages: results from a factorial survey experiment among German employers. J. Int. Migr. Integr. pp 1–23 (2018)
NEGOTIATE: NEGOTIATE Employer Survey. Scientific Use File. Data Documentation, OsloMet, Oslo (2020)
Nunley, J.M., Pugh, A., Romero, N., Seals, R.A.: The effects of unemployment and underemployment on employment opportunities: results from a correspondence audit of the labor market for college graduates. ILR Rev. 70(3), 642–670 (2017)
Oberholzer-Gee, F.: Nonemployment stigma as rational herding: a field experiment. J. Econ. Behav. Organ. 65(1), 30–40 (2008)
Oesch, D., Lipps, O., McDonald, P.: The wage penalty for motherhood: evidence on discrimination from panel data and a survey experiment for Switzerland. Demogr. Res. 37(1), 1793–1824 (2017)
Pager, D., Quillian, L.: Walking the talk? what employers say versus what they do. Am. Sociol. Rev. 70(3), 355–380 (2005)
Pedulla, D.S.: Penalized or protected? gender and the consequences of nonstandard and mismatched employment histories. Am. Sociol. Rev. 81(2), 262–289 (2016)
Pedulla, D.S.: How race and unemployment shape labor market opportunities: additive, amplified, or muted effects? Soc. Forces 96(March), 1477–1506 (2018)
Petersen, T., Togstad, T.: Getting the offer: sex discrimination in hiring. Res. Soc. Stratif. Mobil. 24(3), 239–257 (2006)
Petzold, K.: The role of international student mobility in hiring decisions. A vignette experiment among German employers. J. Educ. Work 30(8), 893–911 (2017)
Petzold, K., Wolbring, T.: What can we learn from factorial surveys about human behavior?: a validation study comparing field and survey experiments on discrimination. Methodology 15(1), 19–28 (2019)
Protsch, P., Solga, H.: How employers use signals of cognitive and noncognitive skills at labour market entry: insights from field experiments. Eur. Sociol. Rev. 31(5), 521–532 (2014)
Protsch, P., Solga, H.: Going across Europe for an apprenticeship? a factorial survey experiment on employers’ hiring preferences in Germany. J. Eur. Soc. Policy 27(4), 387–399 (2017)
Quillian, L., Heath, A., Pager, D., Midtbøen, A.H., Fleischmann, F., Hexela, O.: Do some countries discriminate more than others? evidence from 97 field experiments of racial discrimination in hiring. Sociol. Sci. 6, 467–496 (2019)
Riach, P.A., Rich, J.: Deceptive field experiments of discrimination: are they ethical? Kyklos 57(3), 457–470 (2004)
Rossi, P.: Vignette analysis: uncovering the normative structure of complex judgments. In: Merton, R., Coleman, J., Rossi, P. (eds.) Qualitative and Quantitative Social Research: Papers in Honor of Paul F. Lazarsfeld, pp. 176–186. Free Press, New York (1979)
Sauer, C., Hinz, T., Auspurg, K., Liebig, S., Hinz, T., Liebig, S.: The application of factorial surveys in general population samples: the effects of respondent age and education on response times and response consistency. Surv. Res. Methods 5(3), 26 (2011)
Schwarz, N.: Self-reports: how the questions shape the answers. Am. Psychol. 54(2), 93–105 (1999)
Shamon, H., Dülmer, H., Giza, A.: The factorial survey: the impact of the presentation format of vignettes on answer behavior and processing time. Sociol. Methods Res. (2019). 10.1177/0049124119852382 DOI: 10.1177/0049124119852382
Shi, L.P., Imdorf, C., Samuel, R., Sacchi, S.: How unemployment scarring affects skilled young workers: evidence from a factorial survey of Swiss recruiters. J. Labour Mark. Res. 52(1), 7 (2018)
Sniderman, P.M.: Some advances in the design of survey experiments. Ann. Rev. Polit. Sci. 21(1), 259–275 (2018)
Todesco, L.: The factorial survey in social research: its pros and cons. Adv. Sociol. Res. 23, 1–29 (2017)
Van Beek, K.W.H., Koopmans, C.C., van Praag, B.M.S.: Shopping at the labour market: a real tale of fiction. Eur. Econ. Rev. 41, 295–317 (1997)
Van Belle, E., Caers, R., De Couck, M., Di Stasio, V., Baert, S.: Why are employers put off by long spells of unemployment? Eur. Sociol. Rev. 34(6), 1–17 (2018)
Van Belle, E., Caers, R., De Couck, M., Di Stasio, V., Baert, S.: The signal of applying for a job under a vacancy referral scheme. Ind. Relat. (Berkeley) 58(2), 251–274 (2019)
Wallander, L.: 25 years of factorial surveys in sociology: a review. Soc. Sci. Res. 38(3), 505–520 (2009)
Williams, R.: Using the margins command to estimate and interpret adjusted predictions and marginal effects. Stata J. 12(2), 308–331 (2012)
Wilson, A.: A silver lining for disadvantaged youth on the apprenticeship market—an experimental study of employers’ hiring preferences. J. Vocat. Educ. Train (2019). 10.1080/13636820.2019.1698644 DOI: 10.1080/13636820.2019.1698644
Zschirnt, E., Ruedin, D.: Ethnic discrimination in hiring decisions: a meta-analysis of correspondence tests 1990–2015. J. Ethn. Migr. Stud. 42(7), 1115–1134 (2016)