2.1. MULTEXT-East morphosyntactic resources
The morphosyntactic resources consist of three layers:
-
The morphosyntactic specifications, which set out the grammar and vocabulary of
morphosyntactic features and descriptions. They specificy what, for each language, is the
set of morphosyntactic descriptions (MSDs) and what features they correspond to, e.g. they
specify that the MSD Ncnp is valid for English and maps to the feature structure
Noun, Type:common, Gender:neuter, Number:plural.
-
The morphosyntactic lexicons, which can contain either the full inflectional paradigms
of selected lemmas or entries for corpus attested word-forms. Each entry gives the
word-form, its lemma and MSD, e.g.,
clocks clock Ncnp
-
The morphosyntactically annotated ‘1984’ corpus, where each word is assigned its
context disambiguated MSD and lemma, e.g.,
<w lemma="clock" ana="#Ncnp">clocks</w>
In the next sections we detail each of these layers in turn.
2.1.1. Morphosyntactic specifications
Languages: Bulgarian, Croatian, Czech, English, Estonian, Hungarian, Macedonian, Persian,
Polish, Resian, Romanian, Russian, Serbian, Slovak, Slovene, Ukrainian.
The syntax and semantics of the morphosyntactic descriptions (MSDs) are given in the MULTEXT-East morphosyntactic specifications, which
have been developed in the formalism and on the basis of specifications for six Western
European languages of the MULTEXT project and in cooperation with EAGLES, the Expert Advisory Group on
Language Engineering Standards.
Originally, these specifications were released as a report of the MULTEXT-East project but
have been since extensivelly revised. Nevertheless, the specifications are still structured
as a report, and contain introductory chapters, followed by the list of defined categories
(parts-of-speech), and then, for each category, a table of attribute-values, and the
languages the features are appropriate for. These so called common tables are followed by
language particular sections. Each language section is further subdivided, and can contain
feature co-occurrence restrictions, examples, notes, and full lists of valid MSDs, as well as
localisation information.
The complete specifications are an XML document, encoded as a TEI P5 schema. Further information about these
specifications is available in their TEI
header.
In the distribution, the specifications are given in the source TEI P5 encoding and in
several derived formats:
-
msd/html/
- The specifications as HTML
-
msd/tables/
- Conversion tables for MSDs in various formats
-
msd/xml/
- The specifications in the source TEI XML
-
msd/xslt/
- XSLT scripts for various conversions over the specifications. They work, after a
fashion, but could definitelly be better coded and documented...
2.1.2. Lexicons
Languages: Bulgarian, Czech, English, Estonian, Hungarian, Macedonian, Persian, Polish,
Resian, Romanian, Russian, Serbian, Slovak, Slovene, Ukrainian.
The MULTEXT-East morphosyntactic lexicons have a simple structure, where each lexical entry
is composed of three fields:
-
the word-form, which is the inflected form of the word, as it appears in the text,
modulo sentence-initial capitalisation;
-
the lemma, which is the base-form of the word;
-
the MSD, i.e., the morphosyntactic description of the word-form.
The sizes of the MULTEXT-East lexica vary considerably between the langauges. A few are
quite limited, and serve more as a proof of concept and to assign lexical entries to the
MSDs, but most contain over 20,000 lemmas and can serve as medium sized morphological lexica
for the languages. In addition to explicating the inflectional behaviour of the most common
(and, typically, morphologically the most complex) words of the languages, the lexica also
served to establish the definitive set of valid MSDs for the languages.
The lexica themselves are available in the directory lex/,
where there is also a README file and (note that WWW access to this directory is restricted
to licence holders).
2.1.3. Linguistically annotated ‘1984’
Languages: Bulgarian, Czech, English, Estonian, Hungarian, Macedonian, Persian, Polish,
Romanian, Serbian, Slovak, Slovene.
The novel "1984" by George Orwell is the central component of the MULTEXT-East corpus. This
parallel corpus contains the novel in the English original (about 100,000 words in length),
and its translations into a number of languages.
This corpus, also called the cesAna corpus, contains word level markup, namely context
disambiguated lemmas and MSDs, e.g., <w lemma="it" ana="#Pp3ns">it</w>.
Each MSD is linked to a TEI feature-structure library (automatically derived from the
specifications, and written in the back matter of the novel), which gives for each MSDs its
decomposition into features, e.g.
<fs xml:id="Pp3ns" xml:lang="en" feats="#P0. #P1.p #P2.3 #P3.n #P4.s"/>
and,
in the feature library:
<f name="CATEGORY" xml:id="P0." xml:lang="en"><symbol value="Pronoun"/></f>
<f name="Type" xml:id="P1.p" xml:lang="en"><symbol value="personal"/></f>
<f name="Person" xml:id="P2.3" xml:lang="en"><symbol
value="third"/></f>
...
The translations of ‘1984’ are sentence aligned with the English original, with
hand-validated alignments. From these en-xx alignments the other bi-lingual and one
multilingual alignments were produced automatically. The sentence alignments are stand-off,
i.e. encoded in separate files with pointers to the aligned sentence identifiers. The
alignments are encloding as TEI link
groups, containing links of the form:
<link n="1:2" targets="oana-bg.xml#Obg.1.1.6.7 oana-ro.xml#Oro.1.2.7.7
oana-ro.xml#Oro.1.2.7.8"/>
The corpus is further documented in its TEI headers.
Component files are available in the directory ana/ (note that
WWW access to this directory is restricted to licence holders).
2.2. MULTEXT-East cesDoc corpus
Languages: Bulgarian, Czech, English, Estonian, Hungarian, Romanian, Slovene, Lithuanian,
(partial).
This corpus encodes mostly (quite rich) structural information about the component texts. It
consists of two comparable corpora (fiction, news), the parallel ‘1984’ corpus and a
small multilingual parallel speech corpus. The corpus and its components are further
documented in their TEI headers, and also in the original
MULTEXT-East report D2.1 F: Corpus Collection
and Preparation, although the information there is no longer current in all
respects.
The corpus is further documented in its TEI headers. It
corpus is available in the directory crp/, where the driver file
is ‘mte-cesdoc.xml’ (note that WWW access to this directory is restricted to licence
holders).
2.2.1. Comparable corpus components
Languages: Romanian, Slovene, Czech, Bulgarian, Estonian, Hungarian.
The multilingual comparable corpus contains a fiction part and a news part, where the data
is comparable across the languages in terms of the number and size of texts; each of the 12
parts has approx. 100,000 words. The corpus is structurally marked up with over 40 different
elements; however. sub-paragraph markup has not been harmonised across the languages.
2.2.2. Structural ‘1984’ and alignments
Languages: English, Romanian, Slovene, Czech, Bulgarian, Estonian, Hungarian, (Latvian),
Lithuanian, Serbian, Russian.
The multilingual parallel corpus consists of the novel ‘1984’, about 100,000 words in
length. The corpus contains extensive headers and markup for document structure, sentences,
and various sub-sentence annotations, these similar to the comparable corpus, but better
harmonised over languages.
The translations of ‘1984’ are sentence aligned with the English original, with
hand-validated alignments. From these en-xx alignments the other bi-lingual and one
multilingual alignments were produced automatically. The sentence alignments are stand-off,
i.e. encoded in separate files with pointers to the aligned sentence identifiers. The
alignments are encloding as TEI link
groups, containing links of the form:
<link n="1:2" targets="oana-bg.xml#Obg.1.1.6.7 oana-ro.xml#Oro.1.2.7.7
oana-ro.xml#Oro.1.2.7.8"/>
2.2.3. Speech corpus components
Languages: Romanian, Slovene, Estonian, Hungarian, (English, Czech, Bulgarian)
MULTEXT-East produced a small corpus of spoken texts taken from the EUROM-1 speech corpus.
It comprises the translations (from English) of forty short passages of five thematically
connected sentences. This written part of the corpus is included together with the other
parts of the cesDoc corpus.
For four languages, the texts have also been read, recorded and included in the
distribution. The corpus texts contain links to the spoken passages stored as .wav files. The
speech files are, due to their size, stored and distributed in a separate bundle.
The speech corpus and its components are further documented in their TEI headers. The corpus is available in the directory spc/.
