Introduction

This thematic section explores the desirability of replication in historiography (and the humanities at large). In other words, it explores the potential value of doing historical research again, and doing so systematically. We employ lessons learned from replication studies and the replication crisis in other fields. Elsewhere, some of us have reflected more theoretically on replication in the humanities and historiography (Peels 2019; Peels and Bouter 2018a, 2018b, 2021; Peels, Bouter, and van Woudenberg 2019). Here, we put our feet in the mud by actually carrying out two replication studies and reflecting on what we can learn from them. For this purpose, we have chosen John Hedley Brooke’s seminal 1991 Science and Religion: Some Historical Perspectives as a case study.

This introduction is organized as follows: first, we briefly describe the replication crisis as it occurred—and is partially still ongoing—in the biomedical and social sciences. Second, we consider and argue for the possibility and desirability of replication in the humanities, particularly historiography. After that, we briefly clarify why we selected for replication John Hedley Brooke’s 1991 book Science and Religion, specifically its third chapter. We elucidate why we chose to do both a direct and a conceptual replication. Finally, we describe what procedures we used to carry out the two replication studies and look ahead to what is to come in this thematic section.

The Replication Crisis

Since the early 2010s, there has been an ongoing replication crisis in a number of academic disciplines, such as the biomedical sciences (Begley and Ellis 2012), economics, and the social sciences (Open Science Collaboration 2015). The main concern that brought this crisis about was that attempts to carry out original research again often led to different results. The percentage of attempts at replication that led to different results varied from field to field and in fact from subfield to subfield or even topic to topic, but in many cases 60–85% of the studies failed to replicate. Of course, methodological worries had been around longer, but only in 2012 did this lead to a systematic attempt to analyze the problem and find solutions (Pashler and Wagenmakers 2012). The term “replication crisis” can now be found in numerous fields, as, fortunately, can attempts to solve the problems involved (Baker 2016; KNAW 2018).¹ Numerous people are still working on what exactly to take into account in replication studies and how replication studies differ from field to field (see, e.g., Pittelkow et al. 2023).

It is important to note that replication studies come in several varieties. In line with common practices in the field and with our earlier work on replication, let us make a threefold distinction between reproductions, direct replications, and conceptual replications.² Differences between all three pertain to how much a replication study differs from the original study with regards to research protocol and use of new data or sources (Peels and Bouter 2021).³ The differences can be summarized as follows. Reproductions reanalyze existing data sets. Direct replications use new data, but the approach (what in the sciences is called a research protocol) is the same as that of the original study. Conceptual replications use new data as well, but also modify the approach (protocol) of the original study. See Table 1.

Of course, much of the terminology surrounding replication efforts may be foreign to humanities scholars. For example, historians would likely not speak of “datasets” but of “texts” and “sources,” and would use terms like “method” or “approach” rather than “study protocol.” We return to this issue in due course.

The main arguments in favor of carrying out replication studies are quality control and the corroboration of conclusions. Quality control refers to checking if the original study is of good academic quality. Errors in the original set-up or in the way the research was carried out come to mind. However, more often, quality control refers to a lack of reporting of null-results or selective reporting. Corroboration of conclusions refers to an additional assessment aimed at finding out if the conclusions of the original study are supported.

The unexpectedly high rate of studies that could not be successfully replicated has led to awareness that replication may be a highly important tool to improve what we could call the “self-cleaning capacity” of the involved disciplines. Accordingly, the reputation of replication studies in the sciences has increased sharply over the past couple of years.

Replication in the Humanities

Now, as more evidence has accumulated supporting the notion of a replication crisis in the biomedical, natural, and social sciences, scholars have begun to consider whether replication studies in the humanities are also possible and, if so, what their value and limitations may be (Aguinis and Solarino 2019; Kursell et al. 2020). Rik Peels (2019) and Lex Bouter (Peels and Bouter 2018a, 2018b) have argued that replication studies are both possible and desirable in a wide variety of humanistic disciplines, such as parts of anthropology, archaeology, classical literature, history, linguistics, literary studies, philosophy, the study of the arts, and theology. They have qualified this claim in various ways, for instance by pointing out that it is restricted to empirical studies and does not concern purely theoretical studies (that is, it pertains to a posteriori rather than entirely a priori studies). After all, the latter, such as ethics, metaphysics, and epistemology, usually do not work with empirically collected data but with abstract principles and intuitions that then figure into, for instance, thought experiments.

Peels and Bouter have also pointed out that replication studies in the humanities can be valuable in two ways. First, by reanalyzing the original sources and including new sources, replication can increase the trustworthiness of the original findings. Second, attempts at replication can provide crucial insights into the method, background assumptions, and positionality of the researcher (for more, see Derksen et al. 2024). For these reasons, replication is one important way universities can meet their responsibility to take the humanities seriously and enhance progress in them (see also Peels et al. 2019; for an argument that progress can indeed be made even in some of the more theoretical humanities, see Peels 2020). However, now that the debate on replication in the humanities has begun, various objections to either its possibility or desirability have been raised in the literature. Rather than attempting to be exhaustive, let us sketch and discuss three common objections here.⁴

First, responding to the call for increased attention to replication in the humanities by Lex Bouter and Rik Peels (cf. Peels and Bouter 2021), Sarah de Rijcke and Bart Penders argue that replications are unhelpful in the humanities because the differences between the humanities and other scientific disciplines are too large. In their view, the humanities have different quality criteria that do not allow for replication. For example, where the social and biomedical sciences pursue truth, most studies in the humanities pursue meaning. Unlike other disciplines, the humanities allow for many, sometimes contradictory, interpretations alongside one another. Finally, the humanities relate differently to their objects of study. Where research is mostly one-directional in many other disciplines, humanities scholars engage in continued interaction with their objects of study (de Rijcke and Penders 2018; see also Holbrook, Penders, and de Rijcke 2019; Penders, Holbrook, and de Rijcke 2019).

While de Rijcke and Penders point to some relevant differences between the humanities and other disciplines, their objections to replication are not completely convincing. For example, assuming for the moment that meaning and truth are distinct things (rather than there being truths about meaning), some studies in the humanities are indeed aimed at meaning, but many others are aimed at truth. The study by John Hedley Brooke discussed in this thematic section, for example, aims to shed light on the Merton thesis—that is, the thesis that values associated with radical Protestantism fostered practical science. Clearly, the Merton thesis can be either true or false and is usually discussed in that way.⁵ This holds for many other claims and studies in history as well.

The issue of multiple, coexisting interpretations makes replications in history different but does not undermine their value. If many valid interpretations of a historical source or text are possible, it is still valuable to check if a given interpretation is plausible or valid given the evidence available. Hardly any scholar would argue that all interpretations of historical sources are equally plausible, so it is important to weigh the various existing interpretations against each other. Widely accepted criteria for valid interpretations are available in the field. As a result, not all research in history is idiosyncratic.⁶ Moreover, a replication could even come up with a new valid interpretation and thereby move the discussion forward.

Finally, in contemporary historiography, there is quite some discussion of how people’s specific sociocultural location influences the way in which they interpret the past, so that we can never end up with “the one and only” correct rendering of some past event, series of events, or period. Each time will ask new questions (today, we see many new questions being asked, e.g., from postcolonial perspectives) and accordingly reconstruct the past in novel ways. In that sense, there is a kind of back-and-forth in our engagement with the past. Yet, in our view, de Rijcke and Penders’s point about bidirectional engagement with objects of study does not really apply to history. Historians engage with sources mostly in a one-directional fashion, much like most other scholars in many of the humanities. The issue of continued interaction applies more to studies on living human subjects, such as in psychology and sociology.

Another worry is that replication might enforce standards from other disciplines on the humanities. One might thus regard the attempt to extend replication to the realm of historiography as scientistic, in the sense of suggesting that (natural and social) scientific research methods are the only legitimate ones in the academy. Replication and replicability are often associated with procedures like the preregistration of studies, the sharing of data, and the sharing of code. This can give rise to the idea that embracing replications in history would imply adopting similar research procedures that could change the discipline thoroughly.

We reply that increased replicability will indeed lead to some changes in the way studies are written down, such as more documentation on research protocols and methodology. Yet, it need not lead to embracing quantification of research or forcing historical research into unsuitable procedures. Replication in historiography may, and likely does, require its own standards and procedures, which largely are yet to be proposed and discussed. Such standards and procedures can be tailored to the specificities of the humanities or even to historiography in particular.

A change that is unavoidable for replication studies is more clarity on the research protocol. Studies in history do not usually include a separate methodology section or a lot of details on methodology. Some historians also object that they do not have a set methodology and would have problems articulating a historiographic methodology. In reply, we note that recent historiography is more reflective on its methods and the many varieties in which such methods come. Some new developments, like history studies that make use of big data or studies in digital humanities, often do include methodology sections. Also, history studies that do not have an explicit methodology often have an implicit one. Some make use of quantitative methods. Clear examples are studies in economic history and those that estimate demographic developments. Others (perhaps most) are qualitative in nature and consist of interpreting various sources. Interpretive studies can come in many flavors. For example, some work with critical interpretation in line with critical theory, Marxist theory, or feminist theory; others aim at intentional interpretations of texts, attempting to reconstruct the author’s mind (see Carroll 2000). By looking closely at how historians reached their conclusions from the sources, a research protocol can sometimes be reconstructed.

Reconstructing a research protocol or even urging historians to include details of their research protocols need not imply adopting standards foreign to the humanities. Historiographical research protocols need not be as stringent or clearly delimited as research protocols in the social and natural sciences. Interpretive studies can retain their unique nature or use of individual skills by the researcher while being more detailed about how the study was performed. Presenting more details on methodology may also make the discipline more open for interdisciplinarity and newcomers to the field.

A final hesitation approaches the subject in an almost diametrically opposed manner compared to the objections leveled and discussed so far. Some scholars claim that replication studies are not at all new for historiography. Historians have always been reassessing claims put forth by others. They have also regularly and carefully looked at specific sources time and again, in order to check whether conclusions by others held water. For example, they produce scholarly reviews of each other’s work and occasionally return to the same past events and critically assess the relevant publications of previous historians.⁷ In this sense, historians have always done replication studies, or at least something that resembles them fairly closely. So, what is new in the recent quest for replication, if anything at all? Have historians not done and valued this all along?

We agree. Yet, while historians have indeed been doing studies that resemble replication studies to some extent, strictly speaking, replication studies take on a different form. We suggest that by injecting more rigor and “controllability” into replication, unique advantages can be created. Preregistration, for instance, can be instrumental to tackling or mitigating certain biases.⁸ Also, by sticking closely to the research protocol of the original study, replications keep a stricter focus. In other studies where sources or claims are assessed again, changes are often (and mostly implicitly) made to the hypothesis tested, the research protocol, the sources used, or all of these. Keeping these largely the same allows for a more rigorous assessment of the reliability and validity of the original study. Still, we acknowledge that some conceptual replications can resemble traditional forms of comparative studies in history more closely because conceptual replications diverge further from the original study than direct ones and reproductions. We also acknowledge the continuing need for traditional ways of reviewing historical work—ways that go beyond strict replication—since these can discuss whether correct methods have been used to tackle some historical problem. This topic deserves further attention.

Replication in Historiography

Having discussed possible objections, let us now explore in a bit more detail what can be said in favor of replication studies in historiography. It seems historiography can benefit equally from additional quality checks on studies and from the corroboration of conclusions. With regards to quality control, Cumberledge, Smith Jr, and Riley (2023) note the frequent occurrence of quotation errors in leading history journals.⁹ Quotation errors are references that do not support the propositions for which they are cited. As in most disciplines, references in historiographical studies imply that the study referenced supports the claims being made. Since scholarly references are usually trusted to be correct (especially when they are detailed), references are only checked to a limited degree. Cumberledge et al. suggest that the problem of mis-referring can be partly overcome by increased documentation in references, such as the inclusion of proposition-specific page numbers.

As to the importance of the corroboration of conclusions, Anton Howes notes a number of erroneous conclusions that are frequently repeated or assumed in subsequent research in historiography. An example is the often-repeated claim that the British government sent more troops to quell the Luddites in 1812 than to fight the Napoleonic forces in the Peninsular War in 1808. That claim was popularized by Eric Hobsbawm (1964), and its veracity is still frequently, but wrongly, assumed.¹⁰ More falsehoods like this are repeated as well. For instance, the history of science and religion is already ridden with myths that stand in need of being exposed as such (cf. Numbers 2009; Numbers and Kampourakis 2015, which discuss no less than fifty-two such myths, most having some background in past scholarship). An increased number of replication studies could weed out erroneous conclusions or conclusions that do not stand up to closer scrutiny. We even suggest that the more well known and influential a historical study is, the more important it is to replicate it. That brings us to why we choose John Hedley Brooke’s book for replication.

Choice of Primary Source and Methods

We chose John Hedley Brooke’s seminal 1991 book Science and Religion for replication for the following four reasons:¹¹

1. The study rightly can be called a cornerstone study, that is, a study that is frequently cited, influential, and deemed authoritative in its field.¹²

2. The book contains a limited set of clearly formulated hypotheses and theses.

3. Unlike most historical studies, Brooke’s study includes an extensive bibliographical appendix that may render it more easily replicable.

4. The author of the book is still alive and turned out to be willing to help in our reflections on how to do the replication study.

We decided to replicate chapter three of the book, entitled “The Parallel between Scientific and Religious Reform.” We did so for the following three reasons:

1. This chapter is of sufficiently limited scope to allow for replication.

2. This chapter stands out in that it cautiously engages an important and protracted debate of general interest. This debate focuses on the positive or negative roles religions—or rather, specific and often competing religious denominations—have played in the rise of modern science.

3. This chapter lends itself well to both a direct and a conceptual replication, as we show in the two ensuing articles in this thematic section.

The first replication study in this thematic section, led by Hans Van Eyghen, can be characterized as a direct replication. This study uses a research protocol highly similar to that of John Hedley Brooke and investigates Brooke’s own original sources as well as some sources not used by Brooke at the time or unavailable to him, since they appeared later.

The second replication study, led by Rachel S. A. Pear, can be characterized as a conceptual replication. In this study, the original research protocol was slightly altered in that we looked at Jewish responses to Copernican thought rather than the Christian sources investigated by Brooke. In this way, it should be possible to test whether similar patterns in the interaction between scientific and religious reformations as found by Brooke in a Christian context can also be detected when focusing on Judaism.

Our idea was to experiment with both types of replications in order to understand the challenges that would be encountered in each and how they might be overcome. We evaluate these experiences in the reflection article that directly follows the presentation of the main findings of both replication studies in this thematic section. We also could have chosen to perform a reproduction of Brooke’s chapter (i.e., a reiteration of the scholarly work behind the chapter using the very same sources), thus also seeking the aid of the third member of the replication family. Since the direct replication includes all sources used by Brooke (as far as these could be tracked down) and retained the same research protocol, however, a reproduction did not seem to have much added value.

Procedure

We used the following procedure to carry out the replication studies. After an initial exploration of the original study, we contacted the author, John Hedley Brooke. It turned out that Brooke was more than happy to think along and comment on our work. We then decided to set up an advisory board that would give feedback at various crucial junctures of the replication studies. In the course of the project, we shared information for comment via email and met twice online and once in person (at a workshop for the project in June 2023). The board members were John Hedley Brooke; Ab Flipse (Vrije Universiteit Amsterdam), a historian who specializes in the interaction of science and religion; Jeremy Brown, a scholar of Jewish responses to Copernicus and the director of the Office of Emergency Care Research at the US National Institutes of Health; Jessica Roitman (Vrije Universiteit Amsterdam), who works in Jewish studies and is also a historian; and J. Britt Holbrooke (Department of Humanities, New Jersey Institute of Technology), a philosopher who focuses on interdisciplinarity and has been vocal in raising concerns regarding replication studies in the humanities.

Subsequently, we preregistered both studies.¹³ This is still highly uncommon in the humanities. The basic idea of preregistration is that one lays out and publishes online what one considers to be the main research question of both the original study and the replication study as well as the main hypotheses, method, and any other details that matter to the replication study. All of this is done and published before the empirical research is actually carried out. This is primarily meant to avoid all sorts of biases, such as confirmation bias and hindsight bias, that might otherwise steer one in a particular direction in carrying out the research. It also makes one reflect more carefully on exactly what one is doing, how one is doing it, and why one is doing it that way. A final advantage is that others are aware that one is carrying out a replication study of this kind. They may reach out to join forces or exchange ideas, or even give up on their own intended replication study so as to prevent the waste of time and resources (if too many replication studies of the same original study are going on, which is clearly not yet a problem in the humanities).

After carrying out the two replication studies, we presented the initial results at an international and interdisciplinary symposium at the Vrije Universiteit Amsterdam. The original author John Hedley Brooke commented on the studies. Jeremy Brown, who also contributes to this thematic section, shared his thoughts on replication in historiography. And other renowned and younger scholars from science and religion, art history, and other fields joined in the exploration of replication studies in historiography. We took their comments into account, revised the studies in light of them, and present the final results in this thematic section.

Set-Up of This Thematic Section

The set-up of this thematic section is as follows. After this introduction, we first present the direct replication, led by Hans Van Eyghen, and then the conceptual replication, led by Rachel S. A. Pear. The rationale for this will be clear by now: whereas the direct replication only draws in new sources (namely, sources from or on Puritanism), the conceptual replication both draws in new resources (namely, sources from Judaism) and uses a somewhat revised study protocol in that it considers a kind of source Brooke did not. After that, we carefully reflect on these findings and what they mean for the possibilities and limitations of replication studies in historiography. John Hedley Brooke then presents his reflections on the experience of his work being replicated. In doing so, he also provides background to the original study that cannot be learned from the study itself. Subsequently, Jeremy Brown reflects on his experience with the project and explores which studies in historiography, particularly in science and religion, might lend themselves to replication.¹⁴

Division of Labor

Rik Peels and Gijsbert van den Brink conceived the projects described in this article and were involved in all stages of the planning and execution of the research. Peels also drafted the first version of this introductory article. Hans Van Eyghen and Rachel S. A. Pear carried out most of the historical research.