3.1 No unified knowledge database regarding Byzantine sigillography

§ 10 Although around 80,000 seals have been unearthed and preserved in public or private collections worldwide, there is no unified knowledge database regarding Byzantine sigillography. Moreover, there is no interoperable and annotated ground truth dataset of Byzantine seals. However, the organization, integration, and annotation of sigillographic data are challenging since images and textual descriptions detailing the seals are dispersed worldwide.

3.2 Difficulties to homogenize heterogeneous and scattered data

§ 11 As mentioned before, several collections of seals exist worldwide. Each collection is preserved using different methods, recorded using different metadata, and following different standards. Therefore, another drawback may be homogenizing dissimilar and scattered data (descriptions in different natural languages) and images (different formats, resolutions, and views) among several collections. The SigiDoc initiative, the first EpiDoc extension to Byzantine seals based on XML and TEI-compliant, is under development.

3.3 Agreeing on a validation protocol and metrics

§ 12 Ground truth data annotations are humanly provided classifications of the data on which AI algorithms are trained or against which they are evaluated. Since the seals are generally damaged or incomplete, domain experts sometimes disagree in their interpretation and, consequently, in their annotations. Therefore, defining validation metrics to qualify and verify the annotations is vital. Another challenge will be agreeing on a validation protocol and metrics to evaluate the manual annotations (intra- and inter-annotator agreements).

3.4 Legal issues since copyright may protect some images

§ 13 A potential drawback could be legal issues since copyright may protect some images. Legal problems may arise if the protected work is shared, used, or reproduced. We should first ask permission from the copyright holder when dealing with copyrightable works.

3.5 No IT tool capable of guiding or assisting the users in the recognition of iconographic scenes

§ 14 The analysis and interpretation of Byzantine seals is a complex task requiring expert knowledge of Byzantine sigillography and Byzantine Studies. For example, on a damaged seal, the presence of a character with a nimbus, sword, and wings could suggest that it is Saint Michael. Also, a custom (but by no means a rule) in the Byzantine Empire was to represent the saint corresponding to the owner’s name on his seal. Thus, the seal owner representing St. Michael was probably named Michael. Currently, no tool can guide or assist the experts or students of Byzantine studies while analyzing the seals.

3.6 Need to interpret the seal as a whole and not as isolated parts

§ 15 Domain experts generally understand iconographic scenes after identifying several objects and their relations, for example, by recognizing imperial objects in the hands of emperors and empresses, as well as the specific crowns or clothing they wore. Seals bearing religious figures also follow specific iconographic characteristics. Therefore, it is necessary not only to identify each element (such as crosses, crowns, swords, etc.) but also to put them in context and in relation to each other to understand the whole scene in a seal. Even more, the figures and scenes need to be studied, as well as the object relations within a seal (such as the Virgin Mary holding the Child, a patriarchal cross mounted on three steps). In addition, the relations between different seals (such as belonging to the same owner or containing the same figure or inscription) may also be analyzed.

3.7 Advanced natural language processing (NLP) techniques to deal with different languages

§ 16 Another central point in the study of Byzantine seals is the recognition of characters and inscriptions. Most of the preserved seals contain inscriptions in Greek, but there are also writings in Arabic, Armenian, Latin, and Syriac, among others, although in smaller quantities. Processing natural language inscriptions in different languages requires advanced NLP techniques.

3.8 Lack of data for training

§ 17 One common barrier to using deep learning to solve problems is the amount of data needed to train a model. The quality, quantity, and variability of the available images could run into some difficulties when data for training is lacking. Object and character recognition in ancient artworks is challenging since the samples for training neural networks are relatively small. To overcome this drawback, domain adaptation approaches, and other transfer learning techniques can be applied to reuse well-known datasets for training.

3.9 Consider the changes in dignities and functions over time

§ 18 Clustering seals with similar characteristics and/or belonging to owners with the same dignity and function may help the experts analyze them. The study of the similarities and differences among the resulting clusters of seals may formulate new hypotheses in the humanities. For example, grouping seals belonging to the same owner may improve the hypotheses concerning Byzantine prosopography and help reconstitute family networks.

§ 19 However, the Byzantine Empire had a complex system of bureaucracy. During the more than a thousand years of the empire’s existence, several dignities and functions were created and abolished, and many lost or gained prestige. Honorary titles also change with time, creating new titles to single out members of the imperial family. Therefore, it is important to consider the changes in the administration, the administrative geography, the dignities, the functions, or the iconographic motifs over time.

3.10 Consider the seals’ deterioration level

§ 20 Seals are used to authenticate letters and documents. Generally, they were secured so the document could not be opened without damaging the seal. This is the reason why most of the preserved seals are damaged or partially incomplete. As mentioned before, the objects and figures depicted in seals are used to determine the period of their creation. However, when a seal is damaged or partially destroyed, it is difficult to identify its content. Thus, it is not always possible to estimate the inception date.

3.11 An area of fragility in the middle

§ 21 Seals are hollow in their centre to leave a passage for the thread used to attach the seal to the document, thus creating an area of fragility along this line. Several seals are broken in the centre or have missing parts. There are even seals that are divided into two or more parts. An additional challenge arises when the motifs or patterns to be identified are not complete.

3.12 Information not imprinted in off-centre seals

§ 22 Byzantine seals are made using a blank (i.e., a round piece of lead struck with a clamp bearing text and image engraved, usually in reverse, on the two internal sides). Blanks are produced in prepared molds in different but often standardized sizes. If a person wants to strike a seal, it is necessary to find a blank. Sometimes, the engraving is larger than the blank (or the blank is not centred), and part of the message is not imprinted on the seal. Sometimes, the opposite is true: when the blank is larger, a margin appears between the imprinted message and the border of the blank.

4.1 Segmentation and image annotation

§ 27 The first objective of the BHAI project is to create a curated and annotated dataset of Byzantine seals. Byzantine experts manually curated and annotated the images to build ground truth datasets. Building a corpus of annotated images is vital for training and evaluating AI algorithms.

§ 28 Two types of new data are produced from the images collected: manual annotations of the images and transcription of the texts on the seals. The type of annotation on the images depends on the algorithm used and the desired goal:

• Image classification: The annotation is simply a label within a predefined list. In our case, each image is labelled with the estimated date of the seal (e.g., “10th century” or “11th century”), place of provenance, dignities, and functions of the bearer.

• Object detection: Annotation provides the position of objects in the image with a “bounding box,” a rectangle containing the object.

• Instance segmentation: Annotation consists of precisely outlining objects. It is the most precise annotation but the most time-consuming and tedious to perform manually.

We chose instance segmentation because it produces better and more reliable data for training, which can sometimes compensate for the limited data. Using Supervisely (Supervisely OÜ 2024), we obtain a set of points for each annotated object that constitutes the bounding polygon. From the polygons, we automatically produce bounding boxes for object detection. Regarding the labels used, we consider the types of virgins, the type of prayer, and the centuries divided into four parts. Regarding the iconography elements, we consider different types of crosses, nimbus, and persons (such as Jesus, the Virgin Mary, St. Nicholas, etc.). We also annotate objects and body parts (wings, hands, beards, etc.).

§ 29 Figure 1 shows three examples of annotated seals in the Tatış collection: (a) The Virgin Mary surrounded by text, (b) Greek characters and a cruciform monogram, and (c) a stepped patriarchal cross with tendrils and text. Regarding the inscriptions on the seals, we annotated not only all the Greek characters from the Unicode standard, but also the diacritics from Athena Ruby’s font. For some characters, we also consider variations over time, such as the character beta. The letter beta had several highly divergent forms. In the seals, we consider two variants: B and R = βῆτα. It is important to notice that we annotated the monogram as a whole and each character separately. In summary, each annotated object can be exported as a mask for semantic segmentation. In the end, we have (a) the original image, (b) the annotation in JSON format, (c) the human masks for visualization (where each pixel has the colour of the corresponding class), and (d) the machine masks.

Figure 1

Three examples of annotated seals.

4.2 Detecting visual elements using deep neural networks

§ 30 One objective of the BHAI project is to analyze the iconographic scenes (objects and personages). Therefore, we used deep neural networks to detect different types of crosses and cruciform monograms to reach that goal. Christianity was the official religion in Byzantium, and the emblem of Christianity is the cross, which symbolizes the crucifixion of Jesus Christ. Thus, we decided to identify crosses as the first attempt to analyze Byzantine seal content. During the later phase of the Iconoclasm, crosses began to appear massively on monograms. Even after the end of this period, crosses were still preferred for a long time, especially at the hierarchy’s lowest levels. In the early ninth century, seals adorned with potent crosses were often seen. However, patriarchal crosses (often accompanied by tendrils on the sides) are typically found throughout the late ninth and early tenth centuries (Cheynet 2008).

§ 31 Monograms are like puzzles, and the Byzantines loved them. They create monograms by combining the letters from their baptismal or family names. Every letter of a name must be included, with the possibility of repetition and combination between letters. Monograms can be either block or cruciform. In this article, we are interested in cruciform monograms since they are compositions of letters around and on the branches of a Greek cross. Due to the resemblance of these monograms to crosses, we decided to detect them together. Figure 2 presents some preliminary results.

Figure 2

Examples of crosses and cruciform monograms detected in Byzantine seals.

§ 32 We trained neural networks to detect monograms and different types of crosses using YOLOv8 (Jocher, Chaurasia, and Qiu 2023). It is important to note that the algorithm detects those elements even when the seals are damaged. Byzantine seals have different degrees of deterioration. While many seals are in good condition, others are damaged but still interpretable, and others are so broken or harmed that not even experts can read them.

§ 33 Deep learning techniques are an excellent tool for historians, and several works have successfully used them on images of historical elements (Eyharabide et al. 2023; Rahal, Vögtlin, and Ingold 2023; Rezanezhad, Baierer, and Neudecker 2023). Iconographic elements are an interesting application field for the image segmentation domain. Image recognition techniques can be used to automatically identify some visual characteristics, helping sigillographers in the classification or dating of seals.

5.1 Exploring Byzantium via Byzantine monograms

§ 35 Byzantine monograms are often very difficult to interpret, in particular, because there seems to be no rule for the composition and arrangement of the letters. These letter combinations correspond to a creative process that varies drastically depending on the artisan’s inspiration or the owner’s preferences. There are no predefined guidelines for the creation of monograms. Therefore, many monograms are still undecrypted.

§ 36 Artificial intelligence has the potential to enhance the decryption of monograms significantly. However, deciphering monograms can be challenging, mainly because of the absence of a standardized guideline for the structure and placement of the letters. From a computational point of view, monograms constitute a fascinating object to study because they include a certain degree of combinations that must be compared with onomastic lists. Indeed, when it comes to first names, we can refer to the known elements of onomastics, but it is more complicated regarding last names. Therefore, an element of probability must be considered, leading to an analysis of the linguistic characteristics of onomastics (regions, periods). Based on the preliminary results presented in the previous section, we plan to further explore the monograms on seals. The idea is to develop a semi-automatic tool to help experts read monograms.

5.2 Semantic graphs for representing Byzantine sigillographic data

§ 37 Recently, a growing number of works have used AI techniques applied to graphs to study interconnected data (Devi and Kasireddy 2019). These recent works in graph analysis, especially social network analysis, allow us to apply its methods to other data types, such as historical networks. To our knowledge, few works apply graph theory and formal network analysis methods to studying relationships between historical data, such as that of Byzantine seals.

§ 38 The iconography and inscriptions of Byzantine seals offer unique insights into the social and political elites of the Byzantine Empire. In our future work, we propose representing the relationships between the seals in semantic networks, which are well attested but not yet represented as knowledge graphs (KG). The objective is to model the relationships between seals, people, institutions, and functions to represent the common characteristics and the family, professional, religious, and political ties that united them. The idea is to establish a visual representation of seals, highlighting the roles of elites, politics, and religion in the Byzantine Empire. The characters, objects, legends, and inscriptions continue to be vital for knowing dignities, titles, and functions. In addition, we have other sources of information such as diameter, weight, or colour, which provide clues regarding the hierarchy, career progression, professions, selection processes, and, most importantly, the Byzantine elites’ family, social, and political ties.

§ 39 By analyzing the relations among seals, we can find crucial evidence for describing a person’s professional trajectory, analyzing a family tree, or verifying a person’s presence in a specific location or period. Also, the invocations or prayers in the inscriptions are combined with a wide variety of iconography. These elements collectively convey personal devotion and reveal the religious preference within a community.

§ 40 In this context, several graph analysis methods, such as community detection, can be applied to discover seals that share common characteristics and are closely connected. Other metrics like centrality indicators can be used to identify the most influential people or analyze how the seals’ owners are interconnected in the network using information diffusion techniques in graphs. Thus, we aim to offer new perspectives on a research area mainly dominated by more traditional statistical and prosopographic studies and, at the same time, provide a powerful tool for analysis and visualization.

Editorial