General remarks

This section shall support people who have modified the TEI DTD and want to migrate these modifications from SGML to XML, i.e., who want to use the XML-based P4 DTD with equivalent modifications. We begin with some general remarks, then describe an example DTD modification that covers the most important issues, outline a recommended migration procedure and carry out the key step hands-on on the example.

If the elements or content models that the TEI provides don't quite meet the requirements of your project, there is an official esacpe route: you can modify the DTD in a number of well-defined ways and your documents still remain TEI conformant. This involves creating two extension files, setting some parameter entities, possibly defining new elements or redefining existing ones, and making these modifications known to the parser in the DTD subset at the beginning of the document.

Although the process is a lot simpler than it looks at first glance, many people have taken unofficial escape routes, especially the users of the TEI Lite DTD, who would have been required to first switch to a full TEI DTD before applying local extensions. It is admittedly simpler to just open your local copy of teilite.dtd and change a few lines. Only later will you find out why the TEI Guidelines don't advertise this, and one of those moments could be the migration of your customized DTD to XML.

If you are in this situation now, there are three ways to proceed:

Redo your modifications the official way for the P4 DTD, using extension files: find out what is changed in your local copy of the TEI Lite DTD, and create proper extension files for the TEI P4 DTD to the same effect. You will find it is less hard than you thought, and the next migration will be so much easier. You'll find useful advice for this process in the Guidelines, maybe also in the rest of this section. This is the way we would advocate. Redo your modifications as before: find out what you changed in the TEI Lite DTD, and apply the same changes to your local copy of the TEI XLite DTD. We do not advocate this procedure, but it is, of course, a practical possibility. Take a step back: are those modifications really still needed? Maybe they were made to work around a bug of TEI P3, and this bug has gone? Or they were intended for a feature that was never used?

This being said, the rest of this section shall support you in migrating DTD extensions made using the official procedures. So: what types of TEI extensions exist and what is involved in migrating them from SGML to XML? The guidelines know four kinds of modification: TEI Guidelines P4, section 29.1

deletion of elements; renaming of elements; extension of classes; modification of content models or attribute lists.

The first three cases are extremely easy, the fourth item requires more detailed attention. For practical purposes, it can be subdivided into:

redefinition of attribute lists; modification of existing content models definition and integration of new elements (i.e., hanging the new elements into the existing tree)

The following is a short list of some critical issues involved. In the following subsections, we will work through a fictitious example that covers most of these issues.

Case of element and attribute names is important in XML, you have to be consistent now. It is likely that some of the modifications in your existing P3 extension files involved copying (and then probably modifying) pieces of the TEI DTD files ; you should check whether those DTD pieces have changed from P3 to P4. Some people have made modifications to work around problems in the TEI P3 DTD; if they are fixed in P4, the workaround could cause errors (a notorious example is persName). The SGML DTD syntax for element declarations requires two characters of - or O that indicate whether start and end tag are required or can be omitted. These indicators don't exist anymore in XML DTDs and your private DTD snippets need to be modified. The content model for XML elements is more restricted than for SGML elements. We won't go into fine detail, but the following two points deserve attention: The only type of character data is PCDATA, you cannot define CDATA content to bypass the parser. The inclusion exception syntax does not exist in XML DTDs. In SGML, you could specify an element to be legal everywhere within element X and its children in a single line by using the inclusion exception syntax. This is not possible in XML, you have to add X to all content models individually.

A tutorial example

In this subsection, we will do some simple TEI DTD modifications in SGML. This will then serve as a tutorial example for the migration to XML. While working on this example, the main problems in converting DTD extensions should be covered. Not everyone will need everything treated here, and some needs might not be covered, but this should be an easy, hands-on starting point for most projects. As an additional benefit, this small tutorial might induce people to do their modifications the proper way instead of hacking TEI Lite.

Let's assume that five years ago, we wanted a TEI P3 DTD for prose that meets the following extra requirements (these requirements are tutorial examples only, this is no statement on whether they are recommendable TEI practice):

personal names shall be marked with the persName tag (this requires the TEI extensions for names and dates and a workaround for a bug in the P3 DTD). the pb element shall get an extra attribute imageurl that contains an URL for an image of the page; there shall be a new element ps for the postscriptum of letters, containing normal phrase level content; the elements div1 and div2shall be renamed to volume and letter because our source material is a collection of letters organized that way, and we want to keep that structure and make it explicit; an element toDo shall be available everywhere in the text for editorial meta-comments on the ongoing encoding. Therefore, the content shall be CDATA to allow easy typing of element names and entities that are talked about in these notes (in CDATA content tags and entity references are not recognized by the SGML parser).

These requirements can be cast into TEI SGML by creating two files, my_sgml.ent and my_sgml.dtd that look as follows:

]]>

A sample document godot.sgml using these extensions would look like this:

]> John Doe published by the TEI paper original lost after encoding

Dearest Doug, I will come really soon now. Godot PS: Thanks for all the fish is this <hi> really correct tagging?

Encode next letter ]]>

This example shall be migrated to TEI P4 XML in the next two subsections

Suggested migration procedure

Although the following step-by-step list may sound over-protective, this approach is recommended to help you keep a clear head while you are converting DTD and documents. You are switching from P3 to P4, from SGML to XML DTDs, from SGML to XML documents and from SGML to XML parsers at the same time, and it can be difficult to find one's way through these many potential pitfalls.

Pick an interesting test document from your repository and make sure you can parse it as it is (in SGML form) against your current DTD setup (TEI P3 with your extensions). Maybe make up an example like we did above. Set up a parallel SGML parser environment to parse against the TEI P4 DTD. Before you try to parse your sample file, make sure it works with very simple standard TEI files. Now try to parse your sample document against P4 (in SGML mode). Theoretically, this shouldn't be a problem, but the chance is high you are getting errors. Create an intermediate version of your SGML extension files that fix the problems (the cause is most likely that your extensions work around a bug of the P3 DTD that was repaired in P4). Create new XML extension files based on the SGML ones. We will do this for our example in the next subsection. If you want to continue supporting SGML documents, consider making the extension files dual-use (which shall mean: compatible with XML and SGML, see below) so you need to maintain only one set. If you had to make changes for the previous step (moving from P3 to P4 SGML), you have to decide whether the SGML part of your dual-use setup shall be P3 compatible or P4 compatible. Normally, you should be fine using P4 SGML, and this is the assumption in the following text. If you have created dual-use extension files, use them to parse your SGML document against P4 in SGML mode. Don't proceed until they are correct. Make sure that your XML parser is set up properly by parsing a minimal TEI XML document without extensions. Convert the test document to XML, as described elsewhere in this paper. To have a short made-up example makes this step easier as you avoid the extra confusion of errors from a large-scale text migration. It may be necessary to adapt the tools used so they know about additional tags and attributes in your documents (case normalization!). Try to parse your XML-converted test document. Errors that you see could come from the document conversion or from the migrated DTD extensions. Fix them all. When you can parse in XML-mode (hooray!) and you have dual-use extensions, go back and try the SGML parse again. Then try more documents. Now you could also consult the Pizza Chef and get yourself a flat XML DTD for greater convenience.

Migrating the example DTD

Of the procedure recommended above, we will now focus on rewriting the DTD extensions in XML, with the example DTD modification described earlier as a basis. We will be creating the files my_xml.ent and my_xml.dtd

Before we start out, though, a strategic decision has to be made: Shall we burn the bridges and support only XML in the future? The P4 DTD provides mechanisms to parse both XML and SGML, and we can do the same for our customized DTD, if we will need to support SGML in parallel for some while. It takes a little more thought and effort, but in return you get the comfort of a safe transition period. The key to this is the parameter entity TEI.XML that is defined as INCLUDE in XML mode, and as IGNORE otherwise.

In this document, we shall call this dual-use extensions, and in our example, we shall demonstrate both dual-use and pure XML extensions. The mechanism of using the TEI.XML parameter entity for dual-use extensions can also be used to select between character entity sets for SGML and XML; see the relevant section of this document.

One obvious syntactic difference between SGML and XML DTDs are the omitted tag minimization parameters that appear as - and O in SGML element declarations and indicate whether start and end tags need to be present or not. They are superfluous and gone in XML, where minimization is not allowed. The TEI P4 DTD provides and uses parameter entities %om.RO and %om.RR to be used in their place. For SGML parsing, they expand to - - and - O respectively, for XML parsing they expand to nothing. om.RO is used for elements that require only a start tag (mostly empty elements), om.RR is used for elements that require start and end tag (non-empty elements should be defined that way). We can make use of this mechanism for our dual-use DTD extensions; another useful parameter entity is %TEI.XML that will expand to IGNORE for SGML parsing and to INCLUDE for XML parsing. It can be used to create marked sections for entity and element definitions that will only be seen for SGML or XML respectively. If you feel that this discussion is too DTD-technical for you, don't worry. You probably don't need to understand the background if you follow the examples.

So let's go to work:

We suggested above that before thinking about P4 XML, we should make sure that our document can be parsed with the P4 DTD in SGML mode. When doing this for our example, we get a frightening number of errors like nsgmls:my_sgml.dtd:14:45:E: content model is ambiguous: when the current token is the 1st occurrence of "JOIN", both the 1st and 2nd occurrences of "PERSNAME" are possible. Two occurences of persName? It turns out that the workaround that was necessary to use persName with the P3 DTD is no longer needed and causes trouble instead. So we can remove it from the working copy of our modification file my_sgml.ent. A similar problem would have occured if we had used x.globincl to implement our toDo element. Since the inclusion mechanism is gone, the globincl class is gone as well. Instead, TEI P4 has a new class Incl and elements that shall be available everywhere within text need to be added to x.Incl.

Now let's move towards XML: if we first check for consistent case of the element and attribute names in our DTD, we discover that element ps was once written in uppercase and once in lowercase. We decide that lowercase shall be the correct spelling.

Some things are easy: the renaming of div1 and div2 to volume and letter and the ps tag can remain untouched, except that the - - in the definition of ps needs to be either removed or (if we aim at a dual-use DTD) replaced by %om.RR;.

The dual-use decision comes up again with the pb tag. In XML, we can just write an ATTLIST containing only imageUrl and it will be merged with the existing ATTLIST in the TEI DTD files; there is no need to suppress and copy the definition of pb.

For continuing support of SGML, we have to suppress and redefine the element as before. We find it in the TEI DTD files (teicore2.dtd), copy the P4 definition and modify it for our extra attribute, also replacing - - with %om.RR; in the element definition.

The most difficult problem is the toDo tag. For one, the content model CDATA needs to become PCDATA. This means that existing documents will most probably break, but there is no choice. It might be a solution to turn all the toDo content into CDATA marked sections with an automated search and replace as part of the document conversion, or to escape the contained markup to entity references using a similar procedure.

Also, the simple way of allowing toDo everywhere is no longer possible. This could be a good occasion to check how that element is actually used in practice and where it is really needed. A compromise in our example could be to add it to the class Incl that is part of every content model within text. Sometimes, a more complex redefinition of content models could be necessary. If that is your situation, you may want to consult the inofficial TEI document ED W 69, chapter 8 for in-depth coverage.

The first runs with the XML parser result in many warnings because of redefined parameter entities; this is normal. Some syntax correction is required where XML is more strict than SGML: we forgot a semicolon in a parameter entity reference, %paraContent must not be in parentheses while the #PCDATA for toDo has to be. When you don't know how to get rid of an error, it can be useful to browse the TEI DTD files and compare with your own usage.

The cross-check of the dual-use version with the SGML parser exposes a little additional problem: the document now uses the character entities &lt; and &gt; which are predefined in XML, but not in SGML; once discovered this is easily fixed. In the example file, you will find a solution that looks a little complicated but works flawlessly with SGML and XML.

The reworked extension files in the XML-only form look like this:

]]>

In the dual-use form, the DTD extension file comes out a little longer (my_dual.ent is identical to my_xml.ent above):

] ] > ] ] > ]]>

We can easily convert our short test document manually. All that needs to change is the initial XML declaration, the XML-specific parameter entity, the empty-tag syntax for pb and the escaping of the content of toDo. These steps could serve as models for automated conversion of large documents.

]> ... (header omitted) ... Dearest Doug,

I will come really soon now.

Godot PS: Thanks for all the fish is this <hi> really correct tagging? Encode next letter ]]>