Methods & Tools

AntConc is a freeware corpus analysis toolkit widely used in corpus linguistics for the concordance-based examination of textual data. Developed by Laurence Anthony, the software provides a user-friendly suite of analytical functions, including keyword-in-context (KWIC) concordancing, collocation analysis, word frequency and keyword lists, and n-gram extraction, enabling researchers to investigate lexical patterns and usage across single texts or large corpora. AntConc also works well with non-Latin languages such as Arabic or Chinese/Korean.

Project S has used AntConc in the context of almost all text-focused projects, such as A01, B02, and A03, in order to conduct corpus linguistic analysis of the various corpora.

Clustering is an unsupervised machine learning technique that partitions a set of texts (images) into groups, or clusters, such that members within the same cluster exhibit greater similarity to one another than to those in different clusters, according to a defined distance or similarity measure. In the context of textual and document data, clustering commonly operates on vector representations derived from term-frequency statistics or learned embeddings.

Project S has applied clustering in context of several projects such as A01 and A03, both in the phase of exploratory data analysis and for the actual research questions of the projects.

Dharmamitra is a research initiative dedicated to applying natural language processing methods to Buddhist texts preserved in various Asian languages, developed in connection with Buddhist NLP research conducted at UC Berkeley. Its DharmaNexus module, accessible at dharmamitra.org/nexus, provides an interactive database for exploring intertextuality across Buddhist canonical literature in Tibetan, Pali, Sanskrit, and Chinese, allowing users to trace textual parallels, quotations, and citation relationships across traditions and languages. This intertextual analysis is complemented by dedicated tooling for Tibetan, such as the Tibetan Intertext Lab, an end-to-end pipeline for sentence segmentation, embedding-based similarity computation, and corpus-level intertextuality research on Tibetan texts. The underlying language models and datasets are published under the Buddhist NLP organization on Hugging Face, which hosts resources such as sentence-similarity and generative models trained on Buddhist textual corpora in Tibetan, Sanskrit, Pali, and Classical Chinese, supporting tasks ranging from machine translation and OCR to semantic search and text-reuse detection across the Buddhist textual heritage.

Dharmamitra and the various associated tools such as the Tibetan Intertextual Lab have mainly been used in the context of A03.

Exploratory data analysis (EDA) is an approach to examining datasets through summaries, visualizations, and descriptive statistics. It helps researchers understand the structure and quality of their data, identify patterns and anomalies, and develop research questions and hypotheses for further analysis.

Project S uses exploratory data analysis in every project to understand the available data, identify relevant patterns, and inform the subsequent research questions and analyses.

Text mining refers to the process of extracting structured information and meaningful patterns from textual data. Drawing on methods from natural language processing, machine learning, and statistics, text mining encompasses tasks such as information extraction, topic modeling, sentiment analysis, and named entity recognition. Techniques range from traditional approaches based on term-frequency statistics and rule-based parsing to more advanced methods employing deep learning architectures and contextual language representations. Text mining is widely employed in digital/computational humanities, where it facilitates hypothesis generation and large-scale text analysis that would be infeasible through manual review alone.

Project S has applied text mining to help A01 to gain insights into larger parts of its huge Tafsir corpus.

Text reuse detection encompasses a set of computational methods for identifying instances in which textual content has been reused across one or more documents. Approaches typically rely on techniques such as n-gram analysis, string alignment algorithms, and vector-based semantic similarity measures to quantify the degree of overlap between source and target texts. Text reuse detection is widely applied in corpus linguistics, digital humanities, and information retrieval research, where it supports tasks such as tracing textual transmission, identifying source relationships between historical documents, and evaluating originality in scholarly writing. A popular tool for text reuse detection is passim.

Project S has applied text reuse detection to help A01 to identify metaphor-related transmissions across Tafsirs.