Leiden University Scholarly Publications

Heuritsch, J. (2018)

Effects of Indicators on Knowledge Production in Astronomy

Article in monograph or in proceedings

Nine interviews were conducted with astronomers from Leiden University, and a document analysis was performed on relevant institutional (self-) evaluation documents, annual reports, and CVs of the interviewees. The aim was to perform a qualitative study about the relationship between the research behaviour of astronomers and how their science is being evaluated. This study encompassed the funding and publication system as well as the indicators used to measure the scientific output, its quality and the research performance. It sheds light on how astronomers define high-quality research and how they think that creating knowledge of value is encouraged or hampered by the evaluation processes. The research shows that astronomers are realists who define scientific quality on the basis of “truth” and are driven by curiosity. These two factors make up their intrinsic values and motivation to perform Astronomy. Publication pressure, arising from the requirements of “the system”, creates an...Show moreNine interviews were conducted with astronomers from Leiden University, and a document analysis was performed on relevant institutional (self-) evaluation documents, annual reports, and CVs of the interviewees. The aim was to perform a qualitative study about the relationship between the research behaviour of astronomers and how their science is being evaluated. This study encompassed the funding and publication system as well as the indicators used to measure the scientific output, its quality and the research performance. It sheds light on how astronomers define high-quality research and how they think that creating knowledge of value is encouraged or hampered by the evaluation processes. The research shows that astronomers are realists who define scientific quality on the basis of “truth” and are driven by curiosity. These two factors make up their intrinsic values and motivation to perform Astronomy. Publication pressure, arising from the requirements of “the system”, creates an extrinsic motivation to perform. This results in premature publications, low readability and replicability, risk aversion and a focus on quantity rather than quality. Hence, indicators do not merely represent quality, but also co-constitute what counts as good research. While such constitutive effects of indicator use on research behaviour and content are observed, there is no indication that the astronomer's intrinsic values are co-constituted. This gives rise to a discrepancy between what is being measured by indicators and what astronomers define as scientific quality; the so-called "evaluation gap". Findings on constitutive effects and the evaluation gap in Astronomy lays out the conceptual groundwork for further empirical research and for policy advice on alternative evaluation practices and innovative indicators with the aim of bridging the "evaluation gap".Show less

Scientometrics

All authors

Heuritsch, J.

Date

2018-09-11

Title of host publication

STI 2018 Conference Proceedings

Pages

101 - 106

Publisher

Centre for Science and Technology Studies (CWTS)