Building a Custom Repair Subdialogue
This advanced tutorial does not have an associated RAD application.
Current speech recognition technology is unable to guarantee perfect accuracy. Therefore, a method of repairing broken dialogs is necessary. Even human communication sometimes breakdown, but we repair dialogues with a variety of methods. For example, a conversation can be interrupted by a sudden noise. To repair this dialog, one person might say, "excuse me, could you repeat that please"
RAD includes a default repair subdialogue that is similar to the one mentioned above. The repair subdialogue is accessed when the following 2 conditions are met. The speech recognizer is not confident about any of the available vocabulary and "Repair Mode" is turned on.
From the main menu bar of the RAD canvas, Click View->repair default. A subdialogue screen will appear that contains the built-in, default repair subdialogue.
The basics
Select the repair default as directed above. Notice that the repair default quits after 3 attempts: The prompts for the state named "3rd_attempt" says, "Sorry I give up," and is connected to a STOP object.
This type of dialog repair might not be appropriate for your application. So lets make a few changes.
- Change the prompt in the state named "3rd_attempt" to say, "I still didn’t get that."
- Connect the state named "3rd_attemp" to the Exit object named "return:repeat."
This way the system will persistently attempt to repair the dialog.
Saving Subdialogs
Subdialogs in general
There are two important things to consider when saving an application that contains subdialogues. One, when saving an application, it is necessary only to save the main screen. This will automatically include every subdialog on the canvas. Two, explicitly saving a subdialogue screen simply saves that subdialog. You can include the saved subdialog in other applications. So be careful that when you intend to save your whole application, you do so from the main screen.
Saving the Repair subdialog
Saving the default repair subdialogue screen changes your default repair subdialog permanently. Every time you open RAD, this will be your repair default
Saving the Repair subdialog only with the application
Rather than change the default repair subdialogue permanently, you may wish to change it just for the application you are building. To do this simply save your application as you always have and the new default repair is automatically included, while leaving your default repair subdialog unchanged.
Advanced Activity
The following section applies mainly to advanced users and speech researchers!
1. Delete the following states from the default repair subdialogue:
1st_attempt
2nd_attempt
3rd_attempt
Goodbye
2. Delete the old attempts state, replace it with an Action Object
- Re-name the Action Object "reportScore", and add the following code to that new state's OnExit window:
# Make recognizer info available in array called 'state'
upvar #0 $repair(name) state
array set repair $state(raw)
if {[info exists repair(word,1)] && [info exists repair(garbage,1)]} {
puts "Garbage score for [join $repair(word,1)]: $repair(garbage,1)"
}
if { $repair(attempt) == 1 } {
set repair(previousBest) ""
}
This code sets up the variable state as an alias for the variable $repair(name), which occurs in the 0'th level of your program's code (i.e., the Tcl shell scripts that are producing the RAD interface). It then checks for the existence of some values held by that variable. The puts statement will print out the garbage score against which the 1-Best word in the Recognition Results dialogue box is being compared. This value should be a score closer to 0 than the 1-Best word's score, which is why this entry into the default repair subdialogue occurred.
The code then checks whether any previous entry into this repair subdialogue has occurred at this state, and if not sets the repair(previousBest)variable to indicate that fact.
5. Insert a Conditional Object below the reportScore state.
This state has two output ports. Attach an Exit Object to its left output port.
Attach a Generic Object to its right output port.
7. Place the following code the Conditional Object’s ports
In the left output port, all on one line even though it is not that way below:
[expr $repair(score,1) + 50] < $repair(garbage,1) &&
[string match $repair(previousBest) $repair(word,1)] != 1
Note that the entry above should be entered as a single line, with no line break in the middle. Here it was necessary to break the line so that it would fit on the page, but in your code there should be NO line break; in fact, you will get a formatting error message if there is a line break in this entry.
In the right output port, place a:
1
Thus if the word and garbage scores are further apart than 50 points, or if the 1-Best word is not the same as it was previously the path through the left output port will be followed. Else the path through the right output port will be followed.
set repair(previousBest) $repair(word,1)
This will set the variable repair(previousBest) so that next time through this repair state, it can be compared against the current 1-Best word to see if it is possible that the same utterance has been recognized twice in a row. If so, it will be considered correct.
6. In the Generic Object below the right output port enter a prompt that says:
Was that $repair(word,1)?
Then add a connection to this recognizer state and have its left output port recognize "no" and its right output port recognize "yes."
7. Below the output port that recognizes "no":
Place another Exit Object, with its Return Options set to Repeat a prompt that says "Let's start over," and an action like the following:
set repair(previousBest) ""
So if the user did not accept the clarification, then the repair(previousBest)variable is reset to the empty string.
8. Below the output port that recognizes "yes":
Place an Exit Object with its Return Options set to Continue, a prompt that says Okay and an action like the following:
set state(recog) $repair(word,1)
set repair(previousBest) ""
So if the user did accept the clarification, then the recognition of the state in which the error occurred is fixed, and again the repair(previousBest) variable is reset to the empty string.
9. Create a small application to test your custom repair subdialogue, and build and run it.
Under the start object place a Generic Object and a Goodbye Object on the main canvas for the application and connect them in the order listed. In the Generic Object, provide a prompt which asks the user to choose between a list of choices. Build and run the application, then say a word which is not in the list of choices but is close to one of them. You are trying to fall below the garbage threshold for recognition, but not too far below. The program should reject your utterance with a "Sorry" prompt and prompt you a second time. Repeat the same word you used the first time. Now, if the same word as before is recognized as the most probable, the system will ask you if you have said the word it has recognized twice in a row, even though the score has been below the garbage threshold each time. Reply "Yes" and the application will continue; reply "No" and the application should ask you again for the answer. This repair behavior is different from the original repair strategy.
You have accomplished the minimum for this exercise. If you would like to do more, continue with the next few steps.
9. Add the following code to the Action box of the reportScore state:
# Example code to save wave and raw result:
set dir h:/.rad/data/repair
if ![file isdirectory $dir] {
file mkdir $dir
}
set date [clock format [clock seconds] -format %y_%m_%d-%H-%M-%S]
wave write $state(wav) $dir/$date.wav
set fh [open $dir/$date.raw w]
puts $fh "$state(raw)"
close $fh
puts "save_word: [join $repair(word,1)]"
This will save the pertinent repair information to a file, so you can inspect it later.
10. Save your new repair subdialogue
From the Repair Default Screen, select file - > save screen. Click "This screen and nested screens only" because we are only saving the subdialog. Choose a name for your subdialog.
Important:
In order for this new repair subdialogue to be the new Default Repair, you must save the file as.
<root dir>/cslu/toolkit/2.0/apps/rad/repair/default1.rad