diff --git a/NLP_documentation.md b/NLP_documentation.md index 98ed2f0981fd36606dbba8e6cfc90404928742c0..dac4614a0c9cf56b487277522095dac6531c7ca9 100644 --- a/NLP_documentation.md +++ b/NLP_documentation.md @@ -122,36 +122,38 @@ They test these methods on two large annotated Latin corpora, [the Opera Latin c The script pre-processes Latin legal texts in the ROMTEXT database, the Latin text of the _Digest_ and two other general Latin text corpora. It trains word embeddings models on them with `fastText` by keeping the default dimension of 100 for the word vectors. `fastText` requires the corpus to be a list of string sequences. The raw text of each corpus is transformed to a sequence of lemmas with the custom `latin_lemma_text` function defined in `pyDigest.py`. The function is documented in [the corresponding documentation file](https://gitlab.eps.surrey.ac.uk/mr0048/pydigest/-/blob/master/pyDigest_documentation.md#4-latin_lemma_textlist_of_texts-stopwordsnone). For each corpus, the script outputs a lemma text and a `fastText` model trained with `skip`. -1. ROMTEXT > romtext_lemma.txt, romtext_skip.bin +1. ROMTEXT > `romtext_lemma.txt`, `romtext_skip.bin` The script loads the entire ROMTEXT corpus as a txt file copied from the Amanuensis interface as documented in ["Ddf"](https://gitlab.eps.surrey.ac.uk/mr0048/pydigest/-/blob/master/Ddf_documentation.md#1-creation-of-the-base-dataframes). After removing notes in angular brackets, empty lines and bibliographic headings. The ROMTEXT corpus include 1,590,509 word tokens in 39,368 text units - of which 21,055 belong to the _Digest_. -The script creates a list of low frequency words according to the lemmatised text of ROMTEXT. After removing words which appear 5 or less times in the corpus (15,526 unique items in total), the total words in ROMTEXT are 1,568,586, a difference of 21,923 words between the full and the moderated ROMTEXT corpora. It is the smaller moderated ROMTEXT corpus which is used for creating the `fasttext` model. +The script creates a list of low frequency words according to the lemmatised text of ROMTEXT. After removing words which only appear one or two times in the corpus (13,627 unique items in total), the total words in ROMTEXT are 1,573,383, a difference of 17,126 words between the full and the moderated ROMTEXT corpora. It is the smaller moderated ROMTEXT corpus which is used for creating the `fasttext` model. -2. Digest > digest_lemma.txt, digest_skip.bin, digest_cbow.bin +2. Digest > `digest_lemma.txt`, `digest_skip.bin` -The script takes the 21,055 text units from a flat file. The raw text is pre-processesed before training the `fastText` models. The _Digest_ corpus include 805,101 word rokens in 21,055 text units. The script removes low frequency words as defined in the previous step resulting in a moderated Digest corpus of 802,178 words, a difference of 2,923 words between the full and moderated Digest corpora. The `fasttext` model is trained on the smaller moderated Digest corpus. +The script takes the 21,055 text units from a flat file. The raw text is pre-processesed before training the `fastText` models. The _Digest_ corpus include 805,101 word rokens in 21,055 text units. The script removes low frequency words as defined in the previous step resulting in a moderated Digest corpus of 803,465 words, a difference of 1,636 words between the full and moderated Digest corpora. The `fasttext` model is trained on the smaller moderated Digest corpus. -3. LatinISE > latinise_lemma.txt, latinise_skip.bin, latinise_cbow.bin +3. LatinISE > `latinise_lemma.txt`, `latinise_skip.bin` -We also trained `fastText` models on the annotated LatinISE corpus created by Barbara McGillivray.[^11] The corpus is collected from the [LacusCurtius](https://penelope.uchicago.edu/Thayer/E/Roman/Texts/home.html), [Intratext](http://www.intratext.com/LATINA/) and [Musisque Deoque](http://mizar.unive.it/mqdq/public/index) websites. The texts are enriched with metadata containing information as genre, title, century or specific date. The total corpus includes 13,378,555 tokens.[^12] +A `fastText` models was also trained on the annotated LatinISE corpus created by Barbara McGillivray.[^11] The corpus is collected from the [LacusCurtius](https://penelope.uchicago.edu/Thayer/E/Roman/Texts/home.html), [Intratext](http://www.intratext.com/LATINA/) and [Musisque Deoque](http://mizar.unive.it/mqdq/public/index) websites. The texts are enriched with metadata containing information as genre, title, century or specific date. The total corpus includes 13,378,555 tokens.[^12] The corpus is downloaded in the `latin13.txt` file from the project's [repository](https://lindat.mff.cuni.cz/repository/xmlui/handle/11372/LRT-3170). The script filters "Romana antiqua", "Romana classica" and "Romana postclasscia" texts according to the `era` attribute stored in the document headings. Metadata stored between angular brackets are removed, and documents are split to sentences on `PUN` full-stop. LatinISE includes word tokens, POS-tags and lemmas and three columns separated by tabs. The script only takes the lemmas from the third column to create `latinise.txt`. The `latin_lemma_text` function with Burns' `BackoffLatinLemmatizer` is run on `latinise.txt` to improve lemmatisation with an ouput `latinise_lemma_text.txt`. The 903 documents from the three "Romana" eras include 6,670,292 word tokens in 348,053 sentences. -4. Lasla > lasla_skip.bin, lasla_cbow.bin +4. Lasla > `lasla_skip.bin` The `fastText` models of the Lasla corpus are downloaded from the [word embeddings website](https://embeddings.lila-erc.eu/) of the Naples-based [Linking Latin (LiLa)](https://lila-erc.eu/) project running between 2018-2023, led by Professor Marco Passarotti and funded by the European Research Council (ERC-769994). According to a review article by Patrick J. Burns, Lasla's Opera Latina corpus 154 works from 19 authors totalling 1,630,825 words.[^13] Using the ratio of 19.1645 word tokens per sentences in the LatinISE corpus, we estimate that there are 85,096 sentences in Lasla. -From the `fastText` word embeddings [repository](https://embeddings.lila-erc.eu/samples/download/fasttext/), we have only downloaded the binary files with `skip` and `cbow` method and 100 dimensions. +From the `fastText` word embeddings [repository](https://embeddings.lila-erc.eu/samples/download/fasttext/), the binary file with `skip` method and 100 dimensions has been downloaded. #### 5.2 Corpus comaprison and evaluation -`Manual step` > `syn-selection-benchmark-Latin-legal.tsv` +1. Manual step > `syn-selection-benchmark-Latin-legal.tsv` Sprugnoli and her colleagues provide multiple evaluation methods for their word embeddings models trained on Lasla.[^7] They provide an evaluation benchmark on the LiLa word embeddings website. The first column of the tsv evaluation file includes 2,756 test words. The third column includes the synonyms which are checked and approved manually by a Latin expert. The fourth, fifth and sixth columns include additional three words to create a set for a four-way multiple choice question. In a TOEFL-style multiple choice synonym test, the word embeddings model is asked to "pick" the right synonym for the test word from a set of four. A second column is added manually to the evaluation benchmark including semantic neighbours for words in the first column according to a manual inspection of Adolf Berger's _Dictionary of Roman law_.[^14] There are 473 multiple choice sets created for evaluating how the models capture semantic neighbours in the legal domain. +2. `fasttext_003.py` >