Documentation for ts-to-tei.ipynb done.

%% Cell type:markdown id: tags:

# Transform a TEI-XML export from Transkribus into TEI-XML according to the ConDÉ project specifications.

This script may be optimised or partly rewritten. If I rewrite it, this version will be kept as documentation for the building of the ConDÉ corpus.

You are free to use this script for your own corpus. Know however that you may need to adapt your data, or adapt the script to your data.

Transkribus format observations:

* Informations are split into graphical information (description of text zones on the original image by giving coordinates for the pixels which form the corners of the zone) and textual information, and both are linked through identifiers. This may be shortly summarised thus:

```xml

<TEI>

    <teiHeader/>

    <facsimile>

        <zone id="paragid">

            <zone id="lineid"/>

        </zone>

    </facsimile>

    <text>

        <p facs="#paragid">

            <l facs="#lineid">text</l>

        </p>

    </text>

</TEI>

```

* Anything you type within Transkribus will be noted in the graphic description part instead (`/TEI/facsimile/zone`). Yet you are going to more-or-less copy graphical information and transform the full structure of the text: it is therefore more convenient to base oneself on the text and test the types of the zones through the identifiers.

* Transkribus takes all text information at the same level: zones and lines. This script aims to structure the data better.

### Imports and reclarations

%% Cell type:code id: tags:

``` python

import xml.etree.ElementTree as ET

import re

# This is the TEI namespace declaration.

# Since it is the main namespace, no prefix is added.

ET.register_namespace("","http://www.tei-c.org/ns/1.0")

ET.register_namespace("xml","http://www.w3.org/XML/1998/namespace")

```

%% Cell type:markdown id: tags:

### FUNCTION: convert XML data to ConDÉ semantic TEI

Where the actual script takes place.

ConDÉ format specifics (note: attributes will be noted as att. in the XPath expressions, since the *at* sign is used for user identification on GitHub):

* Three levels of text division: `/TEI/text/div[att.type="part"]/div[att.type="chapter"]/div[att.type="section"]`

* The initial Transkribus types (in `//zone/att.subtype`) have been manually changed to resemble the semantic typing I had in mind, through automatic search and replace in a text editor (I used Oxygen). If you need to adapt this script to your own texts, please change zone types where needed (e.g. in `if para.get('subtype') == "titre":...`.

* If your text may be divided into front, body and back matters, the corresponding TEI elements `/TEI/text/(front|body|back)` must be added manually before application of current script around the identified paragraphs.

* Elements `//l[att.subtype="sectionhead"]` did not get this subtype attribute from Transkribus, but from a regular expression search and replace: by identifying the specifics of the section titles, I was able to get most of them by a regular expression. This meant a search and replace had to be made for each sample in the corpus.

%% Cell type:code id: tags:

``` python

def to_tei(xml_entree, chemin_sortie):

    """

    Function reading a TEI-XML file written by Transkribus prepared with zone typing,

    and returning a more semantically encoded TEI-XML file.

    :param xml_entree: As a string, the local path to the Transkribus TEI-XML output to read.

    :param chemin_sortie: As a string, the local path to write the resulting XML file to.

    """

    # Declare a dict. for identifiers.

    transcriptions = {}

    # The lines of these types are not actually part of the text

    # and must be kept apart: they are about the page/codex format

    # and how to find a reference (p. nb., etc).

    types_a_enlever = ['signature', 'page', 'marginalia', 'header']

    # Parse the original XML file.

    entree_tree = ET.parse(xml_entree)

    entree_root = entree_tree.getroot()

    # DEBUT DE L'ECRITURE DU XML DE SORTIE

    # HERE WE START WRITING THE OUTPUT XML FILE.

    # Création de la racine.

    sortie_root = ET.Element("TEI")

    # Make the root.

    sortie_root = ET.Element("{http://www.tei-c.org/ns/1.0}TEI")

    """# Get the teiHeader from a separate file.

    # Otherwise, make it directly yourself before the <text> element.

    # Récupération du modèle de teiHeader depuis un fichier externe.

    fichier_teiHeader = ET.parse('data/teiHeader.xml')

    teiHeader = fichier_teiHeader.getroot()

    sortie_root.append(teiHeader)

    sortie_root.append(teiHeader)"""

    # Faire un dictionnaire avec les identifiants.

    for ligne in entree_root.findall('.//l'):

    # Make a dictionary with identifiers from all //l/@facs.

    for ligne in entree_root.findall('.//{http://www.tei-c.org/ns/1.0}l'):

        cle = ligne.get('facs')

        transcriptions[cle] = str(ligne.text)

    # TRANSFER AND MODIFY THE ORIGINAL IMAGE INFORMATION AND DIVISION.

    # Création des éléments facsimile, un par un.

    for facsimile in entree_root.findall('facsimile'):

        for zone in facsimile.findall('surface/zone[@rendition="TextRegion"]'):

            if zone.findall('zone[@rendition="Line"]'):

    # Make facsimile elements, one by one.

    # For all zones with types corresponding to "form work" (cf earlier:

    # "not actually part of the text and must be kept apart"),

    # get the corresponding text and put it directly in the zone element

    # as //zone/fw.

    for facsimile in entree_root.findall('{http://www.tei-c.org/ns/1.0}facsimile'):

        # We look for any <zone> element that has "Line" for type.

        for zone in facsimile.findall('{http://www.tei-c.org/ns/1.0}surface/{http://www.tei-c.org/ns/1.0}zone[@rendition="TextRegion"]'):

            if zone.findall('{http://www.tei-c.org/ns/1.0}zone[@rendition="Line"]'):

                for ligne in zone.findall('zone[@rendition="Line"]'):

                for ligne in zone.findall('{http://www.tei-c.org/ns/1.0}zone[@rendition="Line"]'):

                    if zone.get('subtype') == "page":

                        a_trouver = "#" + str(ligne.get('identifiant'))

                        fw = ET.Element('fw', subtype='pageNum', place='top')

                        a_trouver = "#" + str(ligne.get('{http://www.w3.org/XML/1998/namespace}id'))

                        fw = ET.Element('{http://www.tei-c.org/ns/1.0}fw', subtype='pageNum', place='top')

                        fw.text = transcriptions[a_trouver]

                        ligne.append(fw)

                    elif zone.get('subtype') == "header":

                        a_trouver = "#" + str(ligne.get('identifiant'))

                        fw = ET.Element('fw', subtype='head', place='top-centre')

                        a_trouver = "#" + str(ligne.get('{http://www.w3.org/XML/1998/namespace}id'))

                        fw = ET.Element('{http://www.tei-c.org/ns/1.0}fw', subtype='head', place='top-centre')

                        fw.text = transcriptions[a_trouver]

                        ligne.append(fw)

                    elif zone.get('subtype') == "marginalia":

                        a_trouver = "#" + str(ligne.get('identifiant'))

                        fw = ET.Element('fw', subtype='marginalia', place='side')

                        a_trouver = "#" + str(ligne.get('{http://www.w3.org/XML/1998/namespace}id'))

                        fw = ET.Element('{http://www.tei-c.org/ns/1.0}fw', subtype='marginalia', place='side')

                        fw.text = transcriptions[a_trouver]

                        ligne.append(fw)

                    elif zone.get('subtype') == "signature":

                        a_trouver = "#" + str(ligne.get('identifiant'))

                        a_trouver = "#" + str(ligne.get('{http://www.w3.org/XML/1998/namespace}id'))

                        fw = ET.Element('fw', subtype='sig', place='bot-right')

                        fw.text = transcriptions[a_trouver]

                        ligne.append(fw)

        # Once the facsimile element has been treated completely,

        # add the result to the root element.

        sortie_root.append(facsimile)

    text = ET.Element('text')

    front = ET.Element('front')

    body = ET.Element('body')

    back = ET.Element('back')

    ancien_front_repris = entree_root.find('text/front')

    ancien_body_repris = entree_root.find('text/body')

    # NOW PREPARE THE TEXT ITSELF.

    chapitre = ET.Element('div', subtype='chapitre')

    section = ET.Element('div', subtype='section')

    # Make elements to contain the text itself.

    text = ET.Element('{http://www.tei-c.org/ns/1.0}text')

    front = ET.Element('{http://www.tei-c.org/ns/1.0}front')

    body = ET.Element('{http://www.tei-c.org/ns/1.0}body')

    back = ET.Element('{http://www.tei-c.org/ns/1.0}back')

    # But front and body into vars.

    ancien_front_repris = entree_root.find('{http://www.tei-c.org/ns/1.0}text/{http://www.tei-c.org/ns/1.0}front')

    ancien_body_repris = entree_root.find('{http://www.tei-c.org/ns/1.0}text/{http://www.tei-c.org/ns/1.0}body')

    # Prepare empty <div> elements for the middle and inner level.

    chapitre = ET.Element('{http://www.tei-c.org/ns/1.0}div', subtype='chapitre')

    section = ET.Element('{http://www.tei-c.org/ns/1.0}div', subtype='section')

    for para in ancien_front_repris.findall('.//p'):

    # Prepare front matter.

    for para in ancien_front_repris.findall('.//{http://www.tei-c.org/ns/1.0}p'):

        if para.get('subtype') == "titre":

            if chapitre.findall('.//p'):

            if chapitre.findall('.//{http://www.tei-c.org/ns/1.0}p'):

                front.append(chapitre)

                del chapitre

                chapitre = ET.Element('div', subtype='chapitre')

                chapitre = ET.Element('{http://www.tei-c.org/ns/1.0}div', subtype='chapitre')

            head = ET.Element('head')

            for ligne in para.findall('.//l'):

            head = ET.Element('{http://www.tei-c.org/ns/1.0}head')

            for ligne in para.findall('.//{http://www.tei-c.org/ns/1.0}l'):

                head.append(ligne)

            chapitre.append(head)

        elif para.get('subtype') == "coutume" or para.get('subtype') == "note_interne":

            paragraphe = ET.Element('p')

            paragraphe.set('subtype', para.get('subtype'))

            paragraphe = ET.Element('{http://www.tei-c.org/ns/1.0}p')

            paragraphe.set('{http://www.tei-c.org/ns/1.0}subtype', para.get('subtype'))

            for ligne in para.findall('.//l'):

                paragraphe.append(ligne)

            chapitre.append(paragraphe)

        ancien_front_repris.remove(para)

    # Once all paragraphs in the <front> element were added,

    # the last chapter still needs to be added to the <text> element.

    front.append(chapitre)

    del chapitre

    chapitre = ET.Element('div', subtype='chapitre')

    chapitre = ET.Element('{http://www.tei-c.org/ns/1.0}div', subtype='chapitre')

    for parag in ancien_body_repris.findall('.//p'):

    for parag in ancien_body_repris.findall('.//{http://www.tei-c.org/ns/1.0}p'):

        # Si c'est un titre, on ferme le chapitre et on l'ajoute au FRONT.

        # If current paragraph is a chapter title, it means closing both

        # current section and current chapter, adding both to the text element,

        # then opening a new section and a new chapter (the latter will get

        # the current title paragraph as its title).

        if parag.get('subtype') == "titre":

            if section.findall('.//p'):

            if section.findall('.//{http://www.tei-c.org/ns/1.0}p'):

                chapitre.append(section)

                del section

                section = ET.Element('div', subtype='section')

                section = ET.Element('{http://www.tei-c.org/ns/1.0}div', subtype='section')

            if chapitre.findall('.//p'):

            if chapitre.findall('.//{http://www.tei-c.org/ns/1.0}p'):

                body.append(chapitre)

                del chapitre

                chapitre = ET.Element('div', subtype='chapitre')

                chapitre = ET.Element('{http://www.tei-c.org/ns/1.0}div', subtype='chapitre')

            head = ET.Element('head')

            for ligne in parag.findall('.//l'):

            head = ET.Element('{http://www.tei-c.org/ns/1.0}head')

            for ligne in parag.findall('.//{http://www.tei-c.org/ns/1.0}l'):

                head.append(ligne)

            chapitre.append(head)

        elif parag.findall('.//l[@subtype="sectionhead"]'):

            if section.findall('.//p'):

        # If current paragraph contains a section title line, close the current section,

        # add it to the current chapter, and start a new one, with the section title line

        # for a title.

        elif parag.findall('.//{http://www.tei-c.org/ns/1.0}l[@subtype="sectionhead"]'):

            if section.findall('.//{http://www.tei-c.org/ns/1.0}p'):

                chapitre.append(section)

                del section

                section = ET.Element('div', subtype='section')

                section = ET.Element('{http://www.tei-c.org/ns/1.0}div', subtype='section')

            # Formation et ajout du titre de section.

            facs = parag.find('.//l[@subtype="sectionhead"]').get('facs')

            head = ET.Element('head')

            # Make and add section title.

            facs = parag.find('.//{http://www.tei-c.org/ns/1.0}l[@subtype="sectionhead"]').get('facs')

            head = ET.Element('{http://www.tei-c.org/ns/1.0}head')

            head.set('facs', facs)

            head.text = parag.find('.//l[@subtype="sectionhead"]').text

            head.text = parag.find('.//{http://www.tei-c.org/ns/1.0}l[@subtype="sectionhead"]').text

            section.append(head)

            # Formation et ajout du paragraphe suivant le titre.

            quote = ET.Element('quote', subtype='coutume')

            paragraphe = ET.SubElement(quote, 'p')

            for ligne in parag.findall('.//l'):

            # The rest of the paragraph goes into a quote element.

            quote = ET.Element('{http://www.tei-c.org/ns/1.0}quote', subtype='coutume')

            paragraphe = ET.SubElement(quote, '{http://www.tei-c.org/ns/1.0}p')

            for ligne in parag.findall('.//{http://www.tei-c.org/ns/1.0}l'):

                paragraphe.append(ligne)

            section.append(quote)

        # Otherwise, the current paragraph is either main text or a note.

        else:

            # If it has a "coutume" type, it will be a quote element.

            if parag.get('subtype') == "coutume":

                quote = ET.Element('quote', subtype='coutume')

                paragraphe = ET.SubElement(quote, 'p')

                for ligne in parag.findall('.//l'):

                quote = ET.Element('{http://www.tei-c.org/ns/1.0}quote', subtype='coutume')

                paragraphe = ET.SubElement(quote, '{http://www.tei-c.org/ns/1.0}p')

                for ligne in parag.findall('.//{http://www.tei-c.org/ns/1.0}l'):

                    paragraphe.append(ligne)

                section.append(quote)

            # If it has a "note_interne" type, it will be main text.

            elif parag.get('subtype') == "note_interne":

                paragraphe = ET.Element('p', subtype='note_interne')

                for ligne in parag.findall('.//l'):

                paragraphe = ET.Element('{http://www.tei-c.org/ns/1.0}p', subtype='note_interne')

                for ligne in parag.findall('.//{http://www.tei-c.org/ns/1.0}l'):

                    paragraphe.append(ligne)

                section.append(paragraphe)

            # If it has a "note_de_note" type, it will be main text.

            elif parag.get('subtype') == "note_de_note":

                paragraphe = ET.Element('p', subtype='note_de_note')

                paragraphe = ET.Element('{http://www.tei-c.org/ns/1.0}p', subtype='note_de_note')

                for ligne in parag.findall('.//l'):

                    paragraphe.append(ligne)

                section.append(paragraphe)

            # If it has a "footnote-continued" type, it will be main text.

            elif parag.get('subtype') == "footnote-continued":

                paragraphe = ET.Element('p', subtype='note_de_note_continuee')

                paragraphe = ET.Element('{http://www.tei-c.org/ns/1.0}p', subtype='note_de_note_continuee')

                for ligne in parag.findall('.//l'):

                    paragraphe.append(ligne)

                section.append(paragraphe)

            # If there is no subtype, it will be a paragraph.

            elif parag.get('subtype') == None:

                paragraphe = ET.Element('p')

                for ligne in parag.findall('.//l'):

                paragraphe = ET.Element('{http://www.tei-c.org/ns/1.0}p')

                for ligne in parag.findall('.//{http://www.tei-c.org/ns/1.0}l'):

                    paragraphe.append(ligne)

                back.append(paragraphe)

        ancien_body_repris.remove(parag)

    # Ne pas oublier d'ajouter la dernière section au chapitre,

    # et le dernier chapitre au body.

    # After all paragraphs have been treated, add the last section

    # and chapter to the body element.

    chapitre.append(section)

    body.append(chapitre)

    # Add all main elements to the text element, then add it to root.

    text.append(front)

    text.append(body)

    text.append(back)

    sortie_root.append(text)

    # WRITE THE OUTPUT INTO A FILE.

    a_ecrire = ET.tostring(sortie_root, encoding="unicode", method="xml")

    ecriture = open(chemin_sortie, "w")

    ecriture.write(a_ecrire)

```

%% Cell type:markdown id: tags:

### Where to call the function

To use previous function, you may change these parameters:

* the local path to the TEI-XML file you wish to transform (.xml),

* the local path for the output file (.xml).

%% Cell type:code id: tags:

``` python

to_tei('/home/erminea/Documents/CONDE/Ruines_export_TS/TAC_TEI_TS/TAC_TEI_TS_tei.xml',

      '/home/erminea/Documents/CONDE/Ruines_export_TS/TAC_TEI_TS/TAC_transf.xml')

to_tei('/local/path/to/original_transkribus_tei_export.xml',

      '/local/path/to/more_semantic_tei_output.xml')

```

%% Output

    IOPub data rate exceeded.

    The notebook server will temporarily stop sending output

    to the client in order to avoid crashing it.

    To change this limit, set the config variable

    `--NotebookApp.iopub_data_rate_limit`.

    Current values:

    NotebookApp.iopub_data_rate_limit=1000000.0 (bytes/sec)

    NotebookApp.rate_limit_window=3.0 (secs)