Migrating Language Resources from SGML to XML: the Text Encoding Initiative Recommendations

TEI MI W02: Strategic considerations in migration of TEI documents from SGML to XML
TEI MI W03: Practical Guide to migration of TEI documents from SGML to XML
TEI MI W04: Technical Checklist for TEI/SGML documents
TEI MI W06: Migration case studies for nine projects: British National Corpus, MULTEXT-East Multilingual Corpus, Corpus of Middle English Prose and Verse, Japanese Text Initiative, Women Writers Project, Thomas MacGreevy Archive, Documenting the American South, Victorian Women Writers Project, and the Thesaurus Musicarum Italicarum.

Two Reports

↑

TEI MI W02: Strategic considerations

← ↑ →

Intended for administrators and project managers:

Motivation, Opportunities, and Challenges
reasons for migrating
Areas of Migration
instances, DTD, catalog, processing environment
General Recommendations
planning, workflow design: allocating resources, automating conversion, verifying results
Special Considerations in Migration
degrees of migration complexity: from easy to forward-looking
Appendix: Potential Impact of Future Versions of the Guidelines on Migration Issues
changes likely to appear in P5

TEI MI W03: the Practical Guide

← ↑ →

Written primarily for the technical staff:

solutions to specific conversion problems
augmented by Migration Case Studies
covers obtaining the XML DTD and modifying the processing environment
bulk devoted to instance conversion:
- recommended workflow
- conversion tools
- conversion of SDATA entities

Migration workflow

← ↑ →

Instance:

convert the documents to well-formed XML
case normalisation
validation

DTD (TEI):

unextended: switch SYSTEM identifier to P4:
<!DOCTYPE TEI.2 SYSTEM "http://www.tei-c.org/P4X/DTD/tei2.dtd" [...]>
extended: convert extensions only
for one file DTD use Pizza Chef
broken TEI: you're on your own..

Instance conversion tools

← ↑ →

SGML to well-formed XML: (o)sx
osx -d OUTPUT_DIRECTORY -xno-nl-in-tag -xlower -xno-expand-external -xno-expand-internal INPUT.sgml
Prettyprinting: xmllint
Case conversion, default attributes: tei2tei.xsl

SDATA entities

← ↑ →

"specific entity references" not available in XML
In SGML: <!ENTITY amacron SDATA "[amacron]">
In XML: <!ENTITY amacron "ā">
Or, simpler, just ā

Three cases:

SDATA maps to Unicode char: tables
Assigning code points from the private use area of Unicode (PUA)
Using TEI markup constructs

Migrating TEI DTD extensions to XML

← ↑ →

If TEI is not enough, DTD can be modified in a number of well-defined ways - the documents are still "TEI conformant"
This modification involves creating two extension files with the parameter entity, element, and attribute re-definitions
Section descibes how to convert them to XML:
- gives some general remarks
- describes a sample DTD modification that covers the most important issues
- outlines a recommended migration procedure and demonstrates the key steps using the example

Conclusions

↑

The corpus sample cases: BNC and MULTEXT-East

← ↑ →

BNC: large, SGML minimization features, validation
MULTEXT-East: varied, conversion from CES (via XCES)
Thursday 27th May 2004 18:25-19:45 Sala A
O33-TW : Morphosyntactic Corpora & Tools
...
MULTEXT-East Version 3 : Multilingual Morphosyntactic Specifications, Lexicons & Corpora

It's now or never..

← ↑ →

The paper has presented the reports of the TEI TF on SGML to XML migration, which provides detailed instructions for migrating TEI P3 (SGML) documents and DTDs to XML TEI P4 (XML). The reports are meant primarily to serve TEI P3 resource holders, however, they are, in the main, relevant for any SGML to XML conversion project.

So, if you have language resources encoded in SGML, go to
http://www.tei-c.org/Activities/MI/