2.1. MULTEXT-East morphosyntactic resources
The morphosyntactic resources consist of three layers:
- The morphosyntactic specifications, which set out the grammar and vocabulary of morphosyntactic features and descriptions. They specificy what, for each language, is the set of morphosyntactic descriptions, MSDs (or, simpler, PoS tags) and what features they correspond to. For example, they specify that the MSD Ncnp is valid for English and maps to the feature structure Noun, Type:common, Gender:neuter, Number:plural.
- The morphosyntactic lexicons, which can contain either the full inflectional paradigms of selected lemmas or entries for corpus attested word-forms. Each entry gives the word-form, its lemma and MSD, e.g.,
clocks clock Ncnp - The morphosyntactically annotated ‘1984’ corpus, where each word is assigned its context disambiguated MSD and lemma, e.g.,
<w lemma="clock" ana="#Ncnp">clocks</w>
In the next sections we detail each of these layers in turn.
2.1.1. Morphosyntactic specifications
The syntax and semantics of the morphosyntactic descriptions (MSDs) are given in the MULTEXT-East morphosyntactic specifications. The specifications have been developed in the formalism and on the basis of specifications for six Western European languages of the EU MULTEXT project from the 1990s' and in cooperation with EAGLES, the Expert Advisory Group on Language Engineering Standards. The first version of these specifications was released as a report of the MULTEXT-East project but have been since extensivelly revised. Nevertheless, the specifications are still structured as a report, and contain introductory chapters, followed by the list of defined categories (parts-of-speech), and then, for each category, a table of attribute-values, and the languages the features are appropriate for. These so called common tables are followed by language particular sections. Each language section is further subdivided, and can contain feature co-occurrence restrictions, examples, notes, and full lists of valid MSDs, as well as localisation information.
The complete specifications are an XML document, encoded according to the TEI Guidelines.
In specifications are maintained on https://github.com/clarinsi/mte-msd, and, for continuity, mirrored here, i.e. on http://nl.ijs.si/ME/V6/msd/
The source TEI encoding is down-converted to several derived formats:
- html
- Readable format
- tables
- MSD tabless, including TEI MSDs as feature structures
- cooked TEI
- Canonical specifications in TEI
- source TEI
- Editable TEI specifications on GitHub
- xslt
- Conversion XSLT (and Perl) scripts on GitHub
2.1.2. Morphosyntactic Lexicons
Languages: Bulgarian, Czech, English, Estonian, Hungarian, Macedonian, Persian, Polish, Resian, Romanian, Russian, Serbian, Slovak, Slovene, Ukrainian.
The MULTEXT-East morphosyntactic lexicons have a simple structure, where each lexical entry is composed of the following fields:
- the word-form, which is the inflected form of the word, as it appears in the text, modulo sentence-initial capitalisation;
- the lemma, which is the base-form of the word;
- the MSD, i.e., the morphosyntactic description of the word-form;
- optional frequency of triplet in some reference corpus.
The sizes of the MULTEXT-East lexica vary considerably between the langauges. A few are quite limited, and serve more as a proof of concept and to assign lexical entries to the MSDs, but most contain over 20,000 lemmas and can serve as medium sized morphological lexica for the languages. In addition to explicating the inflectional behaviour of the most common (and, typically, morphologically the most complex) words of the languages, the lexica also served to establish the definitive set of valid MSDs for the languages.
The lexica are available from the CLARIN.SI repository:
- Erjavec, Tomaž; et al., 2010, MULTEXT-East free lexicons 4.0, Slovenian language resource repository CLARIN.SI, http://hdl.handle.net/11356/1041.
- Erjavec, Tomaž; et al., 2010, MULTEXT-East non-commercial lexicons 4.0, Slovenian language resource repository CLARIN.SI, http://hdl.handle.net/11356/1042.
2.1.3. Linguistically annotated ‘1984’
Languages: Bulgarian, Czech, English, Estonian, Hungarian, Macedonian, Persian, Polish, Romanian, Serbian, Slovak, Slovene.
The novel "1984" by George Orwell is the central component of the MULTEXT-East corpus. This parallel corpus contains the novel in the English original (about 100,000 words in length), and its translations into a number of languages.
This corpus, also called the cesAna corpus, contains word level markup, namely context disambiguated lemmas and MSDs, e.g., <w lemma="it" ana="#Pp3ns">it</w>.
Each MSD is linked to a TEI feature-structure library (automatically derived from the specifications, and written in the back matter of the novel), which gives for each MSDs its decomposition into features, e.g.
<fs xml:id="Pp3ns" xml:lang="en" feats="#P0. #P1.p #P2.3 #P3.n #P4.s"/>
and, in the feature library:
<f name="CATEGORY" xml:id="P0." xml:lang="en"><symbol value="Pronoun"/></f>
<f name="Type" xml:id="P1.p" xml:lang="en"><symbol value="personal"/></f>
<f name="Person" xml:id="P2.3" xml:lang="en"><symbol value="third"/></f>
...
The translations of ‘1984’ are sentence aligned with the English original, with hand-validated alignments. From these en-xx alignments the other bi-lingual and one multilingual alignments were produced automatically. The sentence alignments are stand-off, i.e. encoded in separate files with pointers to the aligned sentence identifiers. The alignments are encloding as TEI link groups, containing links of the form:
<link n="1:2" targets="oana-bg.xml#Obg.1.1.6.7 oana-ro.xml#Oro.1.2.7.7 oana-ro.xml#Oro.1.2.7.8"/>
The corpus is further documented in its TEI headers.
The corpus is available from the CLARIN.SI repository:
2.2. MULTEXT-East cesDoc corpora
Languages: Bulgarian, Czech, English, Estonian, Hungarian, Romanian, Slovene, Lithuanian, (partial).
The so called "cesDoc" corpora encodes mostly (quite rich) structural information about the component texts. The corpora consists of the parallel ‘1984’ corpus, two comparable corpora (fiction, news), and a small multilingual parallel speech corpus. The corpus and its components are further documented in their TEI headers, and also in the original MULTEXT-East report D2.1 F: Corpus Collection and Preparation, although the information there is no longer current in all respects.
The corpus is further documented in its TEI headers.
2.2.1. Structural ‘1984’ and alignments
Languages: English, Romanian, Slovene, Czech, Bulgarian, Estonian, Hungarian, Lithuanian, Serbian, Russian.
The multilingual parallel corpus consists of the novel ‘1984’, about 100,000 words in length. The corpus contains extensive headers and markup for document structure, sentences, and various sub-sentence annotations, these similar to the comparable corpus, but better harmonised over languages.
The translations of ‘1984’ are sentence aligned with the English original, with hand-validated alignments. From these en-xx alignments the other bi-lingual and one multilingual alignments were produced automatically. The sentence alignments are stand-off, i.e. encoded in separate files with pointers to the aligned sentence identifiers. The alignments are encloding as TEI link groups, containing links of the form:
<link n="1:2" targets="oana-bg.xml#Obg.1.1.6.7 oana-ro.xml#Oro.1.2.7.7 oana-ro.xml#Oro.1.2.7.8"/>
The corpus is available from the CLARIN.SI repository:
2.2.2. Comparable corpus components
Languages: Romanian, Slovene, Czech, Bulgarian, Estonian, Hungarian.
The multilingual comparable corpus contains a fiction part and a news part, where the data is comparable across the languages in terms of the number and size of texts; each of the 12 parts has approx. 100,000 words. The corpus is structurally marked up with over 40 different elements; however. sub-paragraph markup has not been harmonised across the languages.
The comparable corpus and its components are further documented in their TEI headers. The ‘cmp’ corpus is available for download in the directory herecrp/.
2.2.3. Speech corpus components
Languages: Romanian, Slovene, Estonian, Hungarian, (English, Czech, Bulgarian)
MULTEXT-East produced a small corpus of spoken texts taken from the EUROM-1 speech corpus. It comprises the translations (from English) of forty short passages of five thematically connected sentences. This written part of the corpus is included together with the other parts of the cesDoc corpus.
For four languages, the texts have also been read, recorded and included in the distribution. The corpus texts contain links to the spoken passages stored as .wav files. The speech files are, due to their size, stored and distributed in a separate bundle.
The speech corpus and its components are further documented in their TEI headers. The corpus is available in the directory spc/.
