Accessibility

§ 17 Accessibility is the area where Wikipedia is so strong that comparisons to it become almost fruitless. Wikipedia is so easily accessible that authors may discount accessibility as an investigative object altogether in studies (Arazy and Kopak 2011, 90). This becomes an important factor when making multidimensional comparisons of Wikipedia to other sources of information, such as databases behind paywalls or textbooks that must be purchased, possibly even in physical form. An economist might express accessibility in terms of cost—the cost of obtaining a given piece of information—and argue that this cost must be taken into account when calculating a user’s optimal information-seeking path. Less-accurate but readily accessible information might be more valuable to a user than more accurate information that is too expensive, in terms of time, money or convenience, for him or her to retrieve easily. Although many undergraduates may have ready access to paywall-protected databases, depending on the resources their schools are able to devote to subscriptions, accessibility should be taken into account when considering the potential value of Wikipedia overall. Here we will not focus on it, so as to keep a clear view of the issues around accuracy and completeness—the areas in which historians have concerns.

Expertise, accuracy and trust

§ 18 Expertise does not just play a role in the creation of content for undergraduates. It also plays a role in evaluating it. User reactions to content differ depending on their expertise in the content area. In a recent study, Lucassen and Schraagen demonstrated that readers with domain expertise and novices establish trust based on different dimensions of information quality. Novices use surface and source features, such as style and perceived quality of references, to establish trust. In contrast, experts tend to weigh accuracy more heavily (Lucassen and Schraagen 2011). Lucassen and Schraagen were able to demonstrate this effect experimentally, and I believe their findings are confirmed by the behavior we see in students and in professors with respect to Wikipedia. Neil Waters’ article describes his almost visceral reaction to errors of fact in his area of expertise (Waters 2007). Sarah Baker writes that while her students understand that faculty does not want them to use Wikipedia as a source, they do not fully understand why (Baker 2012). Lucassen and Schraagen’s work helps us explain both phenomena. Professors like Waters are experts and for them, the dimension of accuracy looms large. Conversely, students are novices. They judge (because they must) based on surface features such as perceived objectivity and representation. It makes sense, then, that professors and students establish different trust relationships to the same content.

§ 19 Lucassen and Schraagen’s work helps us put accuracy in a larger context. We can re-cast the question about Wikipedia’s absolute accuracy with respect to historical names, dates and events as a question about the information quality and trustworthiness of Wikipedia’s content compared to other information sources undergraduates use in information seeking for courses. A 2009 study found that 84% of students use Wikipedia as a source of information for course work. However, course readings remain the most important source of information for them. More students reported using commercial research databases offered by vendors like ProQuest and online library catalogs than reported using Wikipedia (Head and Eisenberg 2009, 18). Wikipedia is part of an overall information search and retrieval space, a reference horizon within which students operate. In order to evaluate the role of Wikipedia as a source of information for undergraduate course work, we should not ask only about accuracy. Instead, we should take accuracy into account along with other dimensions of information quality and ask how Wikipedia stacks up to other sources of information used by undergraduates, especially materials provided for course readings or accessible via online databases at the library. As Jacobson and Mark argue, it is possible to teach students to interact with the materials in this search and retrieval space even in introductory classes (Jacobson and Mark 2000, 257). Looking beyond accuracy to other dimensions of information quality helps us see Wikipedia’s potential role in the active development of critical engagement with information from varied sources..

Narrative versions of the controversy

§ 26 The Investiture Controversy is a long and complicated story that took nearly half a century to work itself out—and even then its conclusion was a compromise. In order to keep the scope of our investigation manageable we will focus primarily on the events of 1075-85 and on the Concordat of Worms in 1122, without trying to cover the intermediate events. We will look at two aspects: simple accuracy and completeness.

§ 27 In this study I looked at 3 major online reference works that should be available to most undergraduates: Wikipedia, NewAdvent.org’s online edition of the Catholic Encyclopedia, and the Academic Edition of the Encyclopedia Britannica. I also looked at 9 different Western Civilization textbooks from different publishers, taking a variety of historical perspectives (Coffin et al. 2011; Frankforter and Spellman 2012; Hunt et al. 2010; Kagan et al. 2010; Kishlansky et al. 2008; Levack et al. 2011; McKay et al. 2011; Sherman and Salisbury 2011; Spielvogel 2012). I want to examine how these online reference works and textbooks stack up against each other, taking into account their different purposes and intended audiences. One problem, of course, is the variation in scale. A scholarly account of Gregory’s career takes over 160 pages to get from Gregory’s election in 1073 to his death in 1085 (Cowdrey 1998, 75-241). Obviously, content like this will be more complete than a textbook entry of one or two pages. Reference works have yet a different purpose and format. Still, I think we should feel free to ask the same questions of all three types of information—course books, online reference works, and scholarly work—as we try to gauge the quality of historical representations for an undergraduate audience.

Accuracy

§ 28 Questions about factual accuracy can be vexed in the medieval period, since our source base is sometimes sparse and quite valid controversies may arise with respect to interpretation. Some facts are relatively easy to evaluate. For example, at the time the controversy started, in 1075-76, Henry IV had not yet been crowned Emperor. Thus, it is inaccurate, because it is anachronistic, to refer to Henry as the Emperor before this time. Yet 6 out of 9 of the textbooks make this error. None of the online reference works do. In this quite narrow—not to say pedantic—sense, the textbooks tend to be less accurate than the online reference works.

§ 29 However, the treatment of "facts" that depend heavily on interpretation are difficult to analyze across versions of a single narrative. The discussion above about the current revision of the position of lay investiture in the conflict between Henry and Gregory is a good example. Before Schieffer’s intervention, sentences like "Gregory promulgated a condemnation of lay investiture in 1075" or "The primary point at issue between Henry and Gregory was the question of lay investiture" would have seemed accurate. As a result of the revisionary work sketched above, however, the first sentence is at least seriously in doubt and the second does not match the results of recent scholarship, so should be regarded as factually inaccurate. In fact, the consensus at this point seems to be that both must be jettisoned (Blumenthal 1988; Miller 2009), although Cowdrey finds a middle way, positing a conditional prohibition of lay investiture in February 1075 that would still make the first sentence incorrect [Cowdrey 1998, 104-107]). Since Western Civilization courses themselves originated only in the 1920s, this revision has been in place for about a third of the period during which they have been taught (Stearns 2003, 12). Where, in our sample, is this finding taken into account?

§ 30 It turns out that is it hardly anywhere to be found. In fact, the only source in our sample that reflects this development in the scholarly world is Uta-Renate Blumenthal’s article in the Encyclopedia Britannica. Blumenthal, as it happens, has been one of the major contributors to the scholarly debate; the Britannica article on the Investiture Controversy is a nice example of the benefits of having research scholars write reference articles in their areas of specialty. The online version of the Catholic Encyclopedia has the older view, hardly surprising since it was written in 1910. We must conclude, then, that our sources are in general not accurate in their description of the main issue that brought king and pope into conflict.

§ 31 But what about the other aspects of the story?

§ 32 In the case of the Wikipedia, its account could be corrected or improved in several areas. It makes the role of investiture central to the conflict, just as do older works of scholarship and the textbooks. The January 1076 letter from Henry to Gregory is quoted before the elevation of Tedald in Milan is mentioned, giving a misleading impression about the sequence of events. The Dictatus Papae is incorrectly characterized as containing a prohibition of investiture. More seriously, a version of the article that was available from June 10 to June 15, 2012 gave the date of the Dictatus as 1076 rather than 1075, the year for which it is inserted in the Register (a look at that version will show that I made the correction myself). The description of the Concordat of Worms could use some work, although following the link to the main article on the Concordat will point the reader to a more expansive account.

§ 33 All in all, the Wikipedia article I reviewed is quite serviceable, though the writing could use a little polishing and there are a couple of flat-out errors.

§ 34 Is this better than the textbooks--even though these are not crowdsourced, but prepared by professionals? None of the textbooks I reviewed reflects recent scholarship. They all make investiture the driving issue between Gregory and Henry, though the Controversy itself may be characterized as a symptom of more fundamental historical conflicts (eg. Coffin et al. 2011, 259). The attribution of a fully-developed, public position against lay investiture to Gregory in 1075 is not the only problem. Some narratives conflate events, blurring the development of Gregory’s position in the years 1075-1080. Spielvogel, for example, quotes a passage from 1078 in a way that implies a date of 1075 (Spielvogel 2012, 288). Other errors include mistaken characterizations of the positions of figures in the controversy. Hunt et al asserts that Gregory thought Henry "was just a layman who had no right to meddle in church affairs" (Hunt et al. 2010, 319), though Gregory’s interactions with Henry make it quite clear that the pope sought a different balance between ecclesiastical and secular powers, not a sudden exclusion.

§ 35 Sometimes these errors are quite serious, as in the assertion in Frankforter and Spellman that "the new pope immediately made his position clear," followed by a summary of this ostensible position based on the Dictatus Papae. But Gregory was elected in 1073 and the Dictatus appears to date from two years later. In fact, the first two years of Gregory’s papacy were an important period of development, marked, tellingly, by a relatively cordial relationship to Henry. McKay et al. starts the narrative by attributing to 1075 the provisions of Gregory’s decrees in 1078 and 1080. The text then asserts that "[i]mmediately, Henry IV in the Holy Roman Empire, William the Conqueror in England, and Philip I in France protested" (which is pure fiction). It goes on to claim that, "when Gregory died in 1085, Henry invaded Italy, capturing Rome" (McKay et al. 2011, 253). In fact, Henry had invaded already in 1081, and Gregory died in exile. As a last example, two of the texts claim, incorrectly, that Hildebrand was a Cluniac monk (Frankforter and Spellman 2012, 264; Coffin et al. 2011, 259; cf. Blumenthal 1988, 114).

§ 36 As far as accuracy alone is concerned, then, the Wikipedia article I reviewed can certainly hold its own in this group. The Western Civilization texts all contain some discrepancies from recent scholarship, and many of them contain conflations that would make it difficult or impossible to reconstruct accurately the sequence of events in the crucial period 1075-1080. Some contain out-and-out errors, among them errors arguably more serious than the ones I identified in the Wikipedia article. Wikipedia is clearly behind Britannica in terms of its inclusion of recent research but it is not so clearly inferior to textbooks in terms of accuracy as one might expect. As we will see in the next section, Wikipedia comes out ahead in terms of completeness, our other comparative dimension.

Completeness

§ 37 We might think of the story about Investiture as a modular narrative; the narrator may choose to include or exclude various elements. The choice of elements affects the dimension of completeness, which we may regard as a dimension of overall information quality. Excluding a particular dimension of the base narrative may not diminish accuracy per se, but it may be regarded as a diminution of completeness. In the case of a complex narrative like the Investiture Controversy, completeness can become quite important.

§ 38 We can measure completeness in our accounts by identifying some elements that tend to appear in some narratives and not in others. The more of these elements a narrative has, the more complete it is. There is obviously a degree of arbitrariness in the identification of these elements, but this approach helps us get beyond dates and events in evaluating and comparing historical narratives.

§ 39 I chose twelve optional aspects of the Investiture narrative to consider. The choice of topics is based on accounts in the scholarly literature, rather than textbooks or summaries in reference works. While certainly at one level arbitrary, the list was made with Milller’s suggestions for further interpretive directions in mind (Miller 2009). Here is the list:

• Hildebrand/Gregory’s relationship with the pataria movement in Milan
• The conflict over the elevation of Tedald to archbishop in Milan
• The resistance in Saxony to Henry’s rule, beginning in 1073
• Episcopal resistance to Gregory’s reforms in Germany, predating the January 1076 conference with Henry IV
• Canossa
• Matilda of Tuscany
• The intercession by Matilda and others between Henry and Gregory
• Gregory’s exile in 1085
• Controversies over investiture in England
• Controversies over investiture in France
• The Concordat of Worms, 1122
• Regional differences in the Worms settlement (the process in Germany differed from that in Burgundy and Italy)

§ 40 Each of these could fill out the context of the Controversy. It will be noted that in some cases I have broken out single elements (Canossa, Worms) into two or more separate elements. For example, it is possible to mention Canossa without mentioning Matilda, and it is possible to mention Matilda without mentioning her role as an intercessor between Henry and Gregory. Each decision to include or exclude changes the story’s range of significance. Other, valid choices could have been made as well; for example, I ignored the anti-King Rudolf of Rheinfelden and the anti-Pope Clement III, the role of Gregory’s Norman allies, and two noblewomen who along with Matilda were in Gregory’s inner circle, the Empress Agnes (mother of Henry IV) and Beatrice of Tuscany. I also ignored the details of actual debates over investiture and disciplinary actions taken by Gregory against bishops whose elevations were examined for irregularities, although these historical actions help us understand the actual state of play as it was perceived by contemporaries.

§ 41 It can be seen at once how the inclusion or exclusion of these elements might affect the overall meaning of the narrative. For example, the pataria were a reform-minded tradesmen’s group in Milan. Excluding them reduces the range of social classes involved in the controversy and de-emphasizes the role of the laity, with a corresponding (and perhaps distorting) emphasis on ecclesiastics as the primary drivers of the reform movement. As Miller says: if we take a Church-centric view, our understanding of historical causality in the period is narrowed: "When causes are posited, they are Great Popes" (Miller 2009, 1574). Leaving out Matilda writes out another form of lay involvement in reform and leaves us with no female protagonists. Leaving out the role of the intercessors at Canossa, among them Matilda and Abbot Hugh of Cluny, reduces the events there to a personal confrontation between Henry and Gregory—better staging but worse history.

§ 42 Here are the results of the survey I undertook. On the left is the narrative element considered, followed by the number of occurrences in textbooks and in the online sources (Britannica, Catholic Encyclopedia, Wikipedia):

§ 43 This table shows a significant amount of "clumping" of narratives in the textbooks. It is easy to see at a glance that some elements, such as Canossa and the Concordat of Worms, are mentioned across the board, while others, such as the controversy around Tedald or Gregory’s death in exile, make frequent appearances. Matilda’s relative absence is a little surprising, as she is a compelling figure in her own right and adds a crucial element of gender balance to the story. Some of the "mentions" counted here only give her name, without giving a sense for this formidable personality. Some elements, such as the impact of the Controversy in England and France, get significantly better coverage in the reference sources online.

§ 44 While this aggregate view can give us a sense for the relative importance of the elements of the Investiture Controversy narrative across different information sources, it does not allow us to compare the strengths of individual sources. One way to measure the completeness of an individual publication is to count the number of elements that are present in it. When we do that, this is the result:

§ 45 By this measure, Wikipedia is significantly more complete than any of the textbooks or the other online reference materials. However, there are some issues that must be addressed in evaluating this result. First, these numbers do not mean that the textbook accounts are in themselves incomplete. The textbooks I surveyed all have a version of the sequence of events from 1075 to 1022 that includes the key events in the conflict. If the Controversy is seen primarily as a conflict between two individuals representing two institutions, then these accounts are sufficient. Another limitation of this approach is revealed in the comparison between the online reference sources themselves. The Britannica article is very compact, accurate and by far the most reflective of current scholarship, but it does leave out context that might be of interest to an undergraduate information-seeker. Canossa is mentioned, but Matilda is not. The broader social and political worlds in Germany and Italy are left aside; the article’s very tight constraints deprive us of the context in which the protagonists moved and acted. The Catholic Encyclopedia article is long and comprehensive. It seems wrong to claim that it is less complete than Wikipedia, and in terms of contextualization it is far richer than the Britannica piece—certainly more so than this table implies. But it is still true that the Wikipedia article covers a lot of territory, and a lot of that territory is quite useful. Certainly it helps expand on the limited accounts in the textbooks.

§ 46 To sum up our findings here, in some important respects Wikipedia is actually superior to the textbooks that we give our classes. It is always possible to find errors in Wikipedia, and, as I have tried to show here, it always will be possible. A Wiki-skeptic could even introduce errors to see how long they last. But if we take a step back from focusing on the accuracy of names, dates and events to look at other aspects of information quality a different picture emerges. Wikipedia begins to look like a valuable part of the overall information space available to undergraduates, one we should recommend and embrace for what it is.

§ 47 In this article we have not particularly emphasized one of Wikipedia’s most obvious strengths; it is free and accessible to anyone with an Internet connection. This is in very sharp contrast to the textbooks we examined or online resources behind paywalls like Britannica. If we had added accessibility into our evaluation, for example by weighting information quality with a factor for accessibility as part of an overall information quality index, our results might have been quite different. For a student in an American college or university, the cost of accessing these different types of sources—course readings, for-pay online resources, or free sources—may not be very different, as their access may be supported by the institution. Accessibility may loom larger for people who do not happen to be students enrolled in institutions with budgets (albeit shrinking) for for-pay online subscription services, or loan programs to subsidize the cost of textbooks. Of course, this was not our point; we wanted to show that Wikipedia can play a valuable role for students with (we hope) ready access to many information sources.. But if we had wished to describe the resources available to all citizens of the online world, Wikipedia suddenly becomes a phenomenon of an entirely different order. This is the one of the most exciting and most truly revolutionary aspects of this remarkable resource.