User Tools

Site Tools


basic_annotation_workflow

Basic Annotation Workflow

Transcribe your text

Transcribe your text in GitDox. Alternatively, transcribe your text into a text file.

At this point, you may follow one of two paths:

  1. Use the Natural Language Processing (NLP) Service online
  2. Process using our NLP tools individually on your local machine

These two paths are outlined in detail in the following sections.

NLP Service Online Workflow

Run the NLP Service on your transcribed text.

If you have not already done so in GitDox, you will generally want to proofread tokenization as part of the NLP Service process as described in the guidelines.

Import the SGML into a spreadsheet.

Rename the existing layers according to the Annotation layer names guidelines. (Not all layers in the guidelines will exist in your file at this point.)

Remove any redundant columns. These may be hi (keep hi@rend); supplied (keep supplied@reason etc.); gap (keep gap@reason etc.).

Add missing information to existing layers. For instance, replace lb and cb placeholders in lb@n and cb@n columns with line and column numbers from original manuscript.

Note: the following steps are a guide to the kinds of work you will be doing. It is organized around the principle of editing one annotation layer at a time. When using the NLP pipeline to annotate everything at once, some editors, however, prefer to correct all annotation layers for one row at the same time and to go through the file row by row.

Create an original text ("orig") layer (You may do this last; you may even find it easier to do this last.)

Create a new or clean up an existing layer for original text in bound groups ("orig_group") (You may do this last; you may even find it easier to do this last.)

Proofread the normalized (norm) layer.

  • You may wish to use Google Refine.
  • You do not need to simultaneously proofread the norm_group layer; we can reconstruct norm_group using the data in norm.)

Reconstruct the norm_group layer.

Proofread the part of speech (pos), lemma (lemma), and morpheme (morph) layers. Part of speech and lemma are annotated on the norm level.

Proofread the language of origin (lang) layer.

  • You may wish to use Google Refine.
  • Coptic SCRIPTORIUM annotates for language of origin on the morph level not the word (norm) level. Be sure the language of origin tags align with the content and span in the morph layer rather than the norm layer.

Add translation, paragraph, and other layers as necessary following the annotation layer names guidelines.

Add Metadata.

Validate the file using the validation Excel Add-in.

Process using our NLP tools individually on your local machine

Tokenizer

Import into spreadsheet

You now have an Excel file with tokenized morphemes aligned with bound groups, normalized morphemes. (If you are working with a Sahidica document, you may have translations and verses as well; with a diplomatic transcription line breaks and column breaks and other manuscript annotations are aligned.)

Proofread the tokenization of the bound groups. Add or delete rows if necessary. You may wish to use Google Refine.

Normalization

Create a normalized bound group layer

Create a morph layer

Ensuring orig and norm layers are the same span

Part of speech tagging and lemmatization. (Read the tagger instructions for parameters to lemmatize.)

Language of origin tagger

Metadata

Validate the file using the validation Excel Add-in.

basic_annotation_workflow.txt · Last modified: 2017/04/25 14:40 by eplatte