Purpose and use of the guidelines

The guidelines form a standard set of practices for NER evaluators in Canada. The guidelines allow for scores to be consistent.

Details are different from the NER used in the United Kingdom. This is to keep within the practices in broadcasting and captioning in Canada. The developer of NER is Dr. Pablo Romero-Fresco. He has reviewed these guidelines. He confirms that the scores will be very similar.

NER scoring interpretation guide – NER ‘errors’

NER measures the accuracy of captions. It compares the experience of the caption viewer to that of the hearing viewer.

Transcripts must be prepared for NER evaluation. An exact transcript and a caption transcript of 10 minutes of content is prepared. The content is a television show that is live-captioned. The NER evaluator compares the two transcripts and scores the captions. The evaluator assigns an error type to any differences. Each error type has a score deducted.

Errors occur when the following happens.

• There is a loss or change of meaning.
• There are words or phrases that interrupt the reading experience. They do not alter the meaning. But the caption reader has to stop to try and understand them.

Error types

There are six error types. Each one has a score assigned to it. The score means a deduction.

1. Correct Edition (CE) has a score of -0.0. A CE is scored when captions are different from the exact audio but the meaning is still the same. There are no interrupting words or phrases.
2. Omission of Main Meaning (OMM) has a score of -0.5. An OMM is scored when captions have lost the main idea in the audio’s independent idea unit. (There is more information on page 2).
3. Omission of Detail (OD) has a score of -0.25. An OD is scored when the captions have lost one or more modifying meaning. This affects the dependent idea unit but not the main idea. (There is more information on page 2).
4. Benign Error (BE) has a score of -0.25. A BE is scored when a caption is incorrect. It interrupts the reading. But the viewer can easily figure out the original meaning. It is shown in the video context or comes from something similar in the real world.
5. Nonsense Error (NE) has a score of -0.5. A NE is scored when a caption is incorrect. The viewer can’t figure out the original meaning. If the meaning of the idea unit is altered or omitted, then OD, OMM or FIE will be scored.
6. False Information Error (FIE) has a score of -1.0. A FIE is scored when the captions make sense but the information is different from the exact transcript. The caption viewer cannot tell that the information meaning is false.

Idea units and errors

NER divides transcripts into “idea units.” In oral communication a single idea can be spread over several sentences. NER mainly deals with comparing meaning. Grammar does not always correspond exactly to the meaning. One sentence may have an idea that modifies another sentence.

There are two types of idea units in NER. There is an independent idea unit. And there is a dependent idea unit.

An independent idea unit is the same as an oral sentence. It is a full message. The following sentence is an example of an independent unit. “The blaze started this morning at the front of the house.” An independent unit is sometimes made up of several dependent idea units.

A dependent idea unit gives information about an independent idea unit. It changes or adds detail to the main idea. It tells about the “when,” “where,” “how,” etc. of an independent idea unit. In the example sentence, the dependent idea units are “this morning” and “at the front of the house.”

“Omission of Main Meaning” is used when the meaning of the message has been lost. It can mean that an independent idea unit has been left out. But the viewer may not know that the message has been lost. An OMM can also mean that a dependent idea unit has been left out. In that case, the message will be meaningless or will not make sense.

“Omission of Detail” does not change the meaning of a sentence as much as an OMM does. An OD only affects dependent idea units. An OD is used when the independent idea unit still makes sense to the viewer.

“Nonsense Errors” and “Benign Errors” interrupt reading. They do not change meaning. They do not apply to idea units. They are scored when a word or phrase is misspelled or garbled. They are scored if the viewer’s reading experience is interrupted because of the error.

• A “Benign Error” is scored when viewers can understand and put together the original words. This happens with missing apostrophes and punctuation. It can also happen with misspelled words that are close to the original.
• A “Nonsense Error” is scored when the word or phrase cannot be understood. An NE interrupts the viewer experience more than a BE.

NER does not consider what is reasonable to expect of a captioner. If clear sounds are not captioned then an error will be scored.

Captioning a show can be hard. People talk fast and there can be other interfering factors. The evaluator should make a note if the show is difficult to caption. This note will not affect the scoring.

The decision tree

A ‘decision tree’ is used to find out which error to apply. The evaluator looks at an idea unit that has differences between the captions and the transcript. The evaluator asks a series of questions that lead to the right error. The evaluator then moves on to the next idea unit with a difference. The same process is applied.

• Question 1. Does the caption give the full meaning of the idea unit without words or phrases that interrupt the reading experience?
• If “yes” to Question 1, score CE. Move to the next idea unit.
• If “no” to Question 1, go to Question 2. Does the caption for this idea unit give false but understandable information?
• If “yes” to Question 2, score FIE. Move to the next idea unit.
• If “no” to Question 2, go to Question 3.a. Does the caption viewer lose the meaning that was available to the hearing viewer?
• If “yes” to Question 3.a. If the lost meaning is the main idea, score OMM.
• If “yes” to Question 3.a. If the lost meaning is a modifying detail, score OD.
If “no” to Question 3.a. There is no score. Go to the next question.
• If “no” to Question 2, go to Question 3.b. Does any word or phrase in the idea unit interrupt the reading experience? (This can be because of misspellings, missing punctuation, etc.).
• If “no” to Question 3.b. There is no score.
• If “yes” to Question 3.b. Go to Question 4. For each such word or phrase, has it already caused a score of OD or OMM?
• If “yes” to Question 4, there is no additional score.
• If “no” to Question 4. Go to Question 5. Can the interrupting word or phrase be understood quickly?
• If “yes” to Question 5, score BE.
• If “no” to Question 5, score NE.

Combining errors

Errors can be combined in NER. A difference in meaning can be an error. Something that interrupts the reading experience can be an error. Both types of error can be scored in a single idea unit.

A decision tree is used for scoring errors. An OD or OMM error is considered first. Then a BE or NE is considered. If the possible BE or NE caused the loss of meaning, then OD or OMM is scored. The interruption error is not scored. To score both is considered a double count of the same error.

For example, if the meaning of a whole idea unit is lost then 1xOMM is scored. But BEs or NEs may also be scored on the same idea unit. This can happen when several words or phrases interrupt the idea unit.

Both an OD or OMM and an NE may be scored in the same idea unit. This happens when an unrelated garbled word is added. This causes a separate issue in meaning.

An NE is a rare score. Usually, an unintelligible word or phrase will affect the meaning of the idea unit. An OD or OMM will be scored instead. But there are exceptions. An idea unit can be fully communicated but still have a garbled word that interrupts. In this case, an NE is the correct score.

The scores for OD and OMM do not combine. An independent idea unit can have one or more dependent idea units. If the meaning of the main idea is lost, there is just 1xOMM. There are no ODs added for loss of details.

Adding up .25 errors to equal more than a .5 error

Multiple ODs may be scored. This can happen even when their total errors count more than the error scored for an OMM. A 3xOD or 4xOD is possible to score within a single independent idea unit. This follows UK practice.

Lists and idea units

Each element in a list is usually seen as a separate dependent idea unit. An omission would be 1xOD. It can be more. It depends on how many units of meaning were lost. Just because the elements are similar, it does not justify a 1xOD score if more than one is left out.

The loss of a detail can affect the meaning of the whole independent idea unit. Just like leaving out a critical ingredient in a recipe could ruin it. The impact could be 1xOMM or even FIE.

Idea units in play-by-play

A series of actions by one player is treated as one independent idea unit. The order of a single play series is also treated as one independent idea unit. Elements within it are dependent idea units. Their omission is scored OD. Multiple ODs are scored if more than one element is omitted. A 1xOMM is scored if the whole sequence is omitted or loses meaning.

FIEs

In NER, error assessment is not dependent on the importance of the topic. The viewer’s priorities are not known. So all subjects are equal. Given this, changes in meaning are still FIEs even if they are not important.

NER does not consider that the viewer might know the statement is false. The caption does not communicate false information if the viewer doesn’t easily understand it. It does not communicate false information if it suggests a wrong meaning but does not do it clearly.

There is a FIE when the captions make sense but differ from the exact audio. A FIE is also scored when the meaning is replaced with another clear but false meaning.

If the result of the caption is confusing then it will not be a FIE. It may be another kind of error.

The context of the program is important. The misinformation may be corrected in an on screen graphic. Or it may be corrected in following captions. Then it is not a FIE. It may be another kind of error.

Sometimes the correct meaning or words are in visuals on the screen. Visuals can include player numbers, a score bug, a visual scene or surrounding text. Video can give information that can be edited from the captions without loss. An example is weather temperatures that are incorrect in the captions but are correct on the screen.

Grammar and punctuation errors

Punctuation

Punctuation should be added to verbatim transcripts. Differences in punctuation should be scored.

Missing question marks are usually scored BE. The viewer can tell from the word order that a question has been asked. But they have to stop for a moment because the “?” is omitted. This is an interruption. A BE should be scored. A BE is also scored if a period replaces a comma and breaks up a sentence.

There are few complete sentences in play-by-play audio. There may only be an “…” or a “-” between phrases. The meaning is usually not affected. The caption reader does not have to pause to understand.

A missing comma that is included in the verbatim transcript will often make no difference in meaning. So CE may be appropriate to score. Misplaced or missing apostrophes often change the meaning of words. Sometimes a missing apostrophe can be understood. Then it is an interruption and is scored BE.

Grammar

Spoken audio does not always follow the rules of grammar. Captions with the same incorrect grammar as the verbatim are not errors. Captions do not correct the grammar of the audio. But an edit that corrects the grammar of the audio and retains the meaning is scored CE.

Grammatical errors created by the captions may change meaning. Or they may cause a pause in the viewer’s reading flow. So they may cause BEs or other errors. If the grammatical error is a common usage that does not interrupt reading or change meaning, it may be CE.

Meaning and sounds

Some sounds – like “Uh” and “Um” – are not meaningful words. If they are omitted it does not affect the meaning. If they are in the verbatim transcript, it is scored CE.

But some non-words are meaningful sounds. Their omission from the captions is meaningful. Sounds like [laughter] or [applause] are part of the hearing experience. They should be in the verbatim transcript. Their omission may be OMM or OD depending on the context. If noted or understood from other captions, the evaluator may score the omission as CE.

When sounds cover speech, the missing words are not part of the hearing experience. They will not be in the verbatim transcript. It is not an error when they are not in the captions.

Throw-away words and quips

Quips and banter may be important to the show experience. Light comments may also be meaningful.
Phrases like “You know,” “Thanks for your report,” or “Thanks, Erika” may be included in the verbatim but add no meaning. Omitting them is CE.

Corrections

Correction of mistakes

Mistakes that are corrected in captions will not be an error. It will be a CE if it is accurate when corrected.

Wrong nouns corrected in an on-screen graphic

Wrong nouns corrected in an on-screen graphic is a BE worth -0.25. The caption is wrong and interrupts the reader. But the viewer can see that and understand it quickly.

Verbatim info is in a graphic but not captions

In some cases, the captions don’t include the verbatim information. But the caption viewer experiences it. This can be through a sports score or a news reader reading a quote. The reading experience is not interrupted. The full meaning is communicated. This is scored a CE.

Identifying people

Speaker identification (ID) is required in the following cases.

• It is hard to identify a speaker through their captions. Speaker IDs are needed.
• Chevrons (>>) are need when a speaker changes. There is an error when they are missing.

With two speakers, only one ID is needed for each. After that, chevrons will identify them. Speakers identified in the captions do not need ID when speaking. But they do need chevrons.

Sometimes it can be hard to know who said which caption. Speaker IDs with chevrons may be required. This can happen when there are multiple speakers, off screen speakers, or fast speaking with a lag in captions.

There are different kinds of ID omissions. Context and meaning are important. And there are a number of possibilities. A decision tree approach can be helpful. The captions may be missing speaker IDs or chevrons. The evaluator asks the following questions to decide the error.

• Question 1. Does the video, even with lag time, make the speaker ID clear?
• If “yes” to Question 1, score BE.
• If “no” to Question 1, go to Question 2. Does the result misidentify the speaker?
• If “yes” to Question 2, go to Question 3. Does the misidentified speaker appear to say something that, in program context, they don’t agree with?
• If “yes” to Question 3, score FIE.
• If “no” to Question 3, score OD.
• If “no” to Question 2, and if the speaker ID is simply unclear but the meaning is communicated, then score OD.
• If “no” to Question 2, and if the loss of speaker ID means the captions lose the main meaning, then score OMM.

Misidentifying a player in play-by-play

If a misidentified player can be figured out easily from the visuals, it is scored a BE. If the visuals do not correct it clearly, it can be a FIE.

Misspelled names

A misspelled name can lead to different scores.

• A misspelling that makes the idea unit meaningless is scored an OD or OMM.
• A correct name that can be understood from the caption is scored a BE. For example “Deniss” for “Dennis” can be understood.
• A name that can’t be understood but the idea unit is still communicated is scored a NE. For example, “DDNnn” for “Dennis” can’t be understood.
• A name known to be someone else’s might be scored FIE. A misspelled name that is an understandable word might also be scored FIE. This happens if the result is a clear misstatement.
• American spellings are normally scored BE. For example the word “honor” for “honour” is fine. The US spelling is acceptable for a name or proper noun. For example, the “Marine Honor Guard” would keep the US spelling.

The name of a famous individual was edited out, but their title remains

An omitted name or title that is critical to the main point is scored an OMM. Omitted names or titles shown in other captions or visuals are scored CE. The meaning may be the same even with omitted words. But without additional sources, the omission would be an OD.

In some shows, the audio may have the last names of reporters or regular guests. But the captions may omit the last names. These omissions are OD unless the last name is in nearby captions or visuals. Any background knowledge of the viewer is not considered in NER.

Preparation of transcripts

Preparing transcripts is the key to scoring errors. It must be done carefully.

• NER requires that 10 minutes of live content be transcribed and scored. It does not need to be continuous. It can bridge over commercials and other pre-captioned material. That material is not scored or included in the transcripts.
• All of the audible words should be correct in the verbatim transcript. If a word is inaudible, it is not part of the hearing experience. So, [inaudible] or a similar phrase can be used to signal that these omissions are not scored.
• Chevrons, speaker IDs and bracketed notes should be in the verbatim transcript. Examples include [applause] and [music].
• Spoken audio has no punctuation. It must be added to the transcript. In general, the captioner’s punctuation can be kept in the verbatim. It can be added to or altered. For example, if the captioner’s punctuation choice has caused a difference in meaning. This might cause an OD, OMM, or FIE error. An interruption caused by missing or incorrect punctuation should be inserted.
• The verbatim transcript should not attempt to correct grammar in the audio. It should not use punctuation to correct written English.
• It can be difficult to know how to correct the grammar. It is best not to add commas to the verbatim transcript unless the meaning requires them. There are classic cases where missing commas can cause a FIE. Sentences like “He eats shoots and leaves.” or “Let’s eat Grandma.” can cause a FIE.
• Quotation marks should be in the verbatim transcript. They are needed if the correct meaning is interrupted for the reader.
• Clear music lyrics that can be head should be in the verbatim transcript. They may be omitted in favour of music symbols or the note [music]. These can be used if the lyrics are masked by dialogue, too fast to be understood, or unclear.
  • Note that a summary or description of the lyrics that captures their meaning is usually CE. Unless the exact words are necessary in context, for example with a national anthem.
• Clear off-camera speech that can be heard should be included in the transcript. There should be a source ID given.
• The verbatim transcript does not need to include sounds and meaningless words like “Uh” or “Um.” But false starts like “I – I said” should be included. Meaningless interjections like “you know” should be included. And standard words like “Thanks” should be included. If these are omitted from the captions then it will probably be scored CE.
• The use of hyphens varies. If they are needed for meaning but omitted then they are an error. They should be in the verbatim transcript.

Caption transcripts need to be prepared carefully. They need to be proofread even if the text comes from a broadcast logger. NER is about the viewer experience. Evaluations must be based on what was seen and heard by the viewer.

The NER score calculation

The NER score is calculated as follows.

NER SCORE = (Words – NER Deductions) / Words x 100

The following definitions are used for the calculation.

• “Words” are the total number of words captioned in the segment that is evaluated.
  • “Words” include chevron groups and punctuation marks. Words in brackets count as a single word. These include captions like [light applause] and [laughter].
• “NER Deductions” are the total of all error deductions. These are assessed by the evaluator of the segment.