NAME

hmmscore.tcl - HMM performance analysis tool

AVAILABILITY

script/hmm_1.0

SYNOPSIS

  hmmscore.tcl reference hypothesis [options]

PARAMETERS

reference

Input reference transcriptions.

hypothesis

Hypothesized transcriptions (output of HMM decoding phase - hmmsearchl.tcl).

OPTIONS

-suppress

A list of words which should be ignored when scoring.

-config

Read command line options (configuration info) from this file. The configuration file is in essence a Tcl script which sets the required internal variables.

 set config(config,suppress)    ""

These values will override the preset default values. All subsequent command line options will override the values specified by the configuration script. Command line parameters are specified using the param variable.

 set param(train,reference) digit.ref
 set param(train,hypothesis) digit.ans

Since command line parameters are typically not optional the user needs to specify the command line parameters as a single "-" character for the settings defined in the configuration script to take effect.

-help

Provides a short description of the command line options.

DESCRIPTION

hmmscore.tcl is used to compute the performance of systems. The performance measures are based on the input reference and hypothesized transcriptions. Both these files must be in the CSLU-HMM transcription file format.

The analysis of sentences is based on the string-alignments computed using Dynamic Programming. A typical output would as follows

 # words        : 4152
 # insertions   : 87 (2.09537572254)
 # deletions    : 43 (1.03564547206)
 # substitutions: 107 (2.57707129094)
 Word Correct    : 96.387283237
 Sentence Correct: 78.859527121
 Accuracy        : 94.2919075145

The analysis computes the percentage word correctly recognized, the insertion, deletions and substitutions. From these the percentage word accuracy is computed as follows:

 Accuracy = %Correct - %Ins

SEE ALSO

CSLU-HMM transcription file, hmmsearch.tcl

AUTHOR

Johan Schalkwyk
Center for Spoken Language Understanding
Oregon Graduate Institute of Science & Technology