References

Up: Ch. 13: Evaluation Previous: 13.10 Character Recognition

Chapter 13 References

A93: E. Arnold et al. Special issue on evaluation of MT systems. Machine Translation, 8(1-2):1--126, 1993.
A94: D. Arnold et al. Machine translation: an introductory guide. NCC/Blackwell, Manchester, Oxford, 1994.
AMT92: MT evaluation: basis for future directions, Washington, D.C., 1992. Association for Machine Translation in the Americas.
ARP93a: Advanced Research Projects Agency. Proceedings of the 1993 ARPA Human Language Technology Workshop, Princeton, New Jersey, March 1993. Morgan Kaufmann.
ARP93b: Proceedings of the Fifth Message Understanding Conference, Baltimore, Maryland, August 1993. Morgan Kaufmann.
ARP94: Advanced Research Projects Agency. Proceedings of the 1994 ARPA Human Language Technology Workshop, Princeton, New Jersey, March 1994. Morgan Kaufmann.
B94a: L. Balkan et al. Test suites for natural language processing. Translating and the Computer, 16:51--58, November 1994. papers presented at a conference.
B94b: F. Bimbot et al. Assessment methodology for speaker identification and verification systems: an overview. Technical Report SAM-A Project 6819, Task 2500, SAM-A, Martigny, Switzerland, 1994.
BAF91: E. Black, S. Abney, D. Flickenger, C. Gdaniec, R. Grishman, P. Harrison, D. Hindle, R. Ingria, F. Jelinek, J. Klavans, M. Liberman, M. Marcus, S. Roukos, B. Santorini, and T. Strzalkowski. A procedure for quantitatively comparing the syntactic coverage of English grammars. In DARPA [DAR91b].
Bai92: H. S. Baird. Document image defect models. In H. S. Baird, H. Bunke, and K. Yamamoto, editors, Structured Document Analysis, pages 1--16. Springer-Verlag, 1992.
BGL93: E. Black, R. Garside, and G. Leech, editors. Statistically-Driven Computer Grammars of English: The IBM/Lancaster Approach. Rodopi, Amsterdam, Atlanta, 1993.
Bla93: E. Black. Parsing english by computer: The state of the art. In Proceedings of the 1993 International Symposium on Spoken Dialogue, Waseda University, Tokyo, October 1993.
Bri92: E. Brill. A simple rule-based part of speech tagger. In Proceedings of the Third Conference on Applied Natural Language Processing, Trento, Italy, March 1992.
CBP94: G. Chollet, F. Bimbot, and A. Paoloni, editors. Proceedings of the ESCA Workshop on Automatic Speaker Recognition, Identification and Verification, Martigny, Switzerland, April 1994. ESCA.
CCD91: G. Chollet, F. Capman, and J. F. A. Daoud. On the evaluation of recognizers---statistical validity of the tests. Technical Report SAM-ENST-02, SAM, 1991.
CCW91: CCW. Research achievements on Chinese character and voice recognition. China Computer World, 349, July 1991. Written in Chinese.
CHA95: R. A. Cole, L. Hirschman, L. Atlas, M. Beckman, A. Bierman, M. Bush, J. Cohen, O. Garcia, B. Hanson, H. Hermansky, S. Levinson, K. McKeown, N. Morgan, D. Novick, M. Ostendorf, S. Oviatt, P. Price, H. Silverman, J. Spitz, A. Waibel, C. Weinstein, S. Zahorian, and V. Zue. The challenge of spoken language systems: Research directions for the nineties. IEEE Transactions on Speech and Audio Processing, 3(1):1--21, January 1995.
CO94: P. R. Cohen and S. L. Oviatt. The role of voice in human-machine communication. In David B. Roe and J. Wilpon, editors, Voice Communication Between Humans and Machines, pages 34--75. National Academy of Sciences Press, Washington, DC, 1994.
Cou66: National Research Council. Appendices 9--15. In Languages and Machines: Computers in Translation and Linguistics. National Academy of Sciences, Washington, DC, 1966.
DAR86: Defense Advanced Research Projects Agency. Proceedings of the DARPA Speech Recognition Workshop, 1986. SAIC-86/1546.
DAR89: Defense Advanced Research Projects Agency. Proceedings of the Second DARPA Speech and Natural Language Workshop, Cape Cod, Massachusetts, October 1989.
DAR90: Defense Advanced Research Projects Agency. Proceedings of the Third DARPA Speech and Natural Language Workshop, Hidden Valley, Pennsylvania, June 1990. Morgan Kaufmann.
DAR91a: Proceedings of the Third Message Understanding Conference, San Diego, California, May 1991. Morgan Kaufmann.
DAR91b: Defense Advanced Research Projects Agency. Proceedings of the Fourth DARPA Speech and Natural Language Workshop, Pacific Grove, California, February 1991. Morgan Kaufmann.
DAR92a: Proceedings of the Fourth Message Understanding Conference, McLean, Virginia, June 1992. Morgan Kaufmann.
DAR92b: Defense Advanced Research Projects Agency. Proceedings of the Fifth DARPA Speech and Natural Language Workshop. Morgan Kaufmann, February 1992.
Eag95: Eagles. Report of the spoken language systems working group 5. Technical report, EAGLES, EAGLES Secretariat, Istituto di Linguistica Computazionale, Via della Faggiola 32, Pisa, Italy 56126, Fax: +39 50 589055, E-mail: ceditor@tnos.ilc.pi.cnr.it, 1995. In press.
Eur93: Eurospeech '93, Proceedings of the Third European Conference on Speech Communication and Technology, Berlin, September 1993. European Speech Communication Association.
F92: A. Fourcin et al. ESPRIT project 2589 (SAM) multi-lingual speech input/output assessment, methodology and standardization. Technical Report SAM-UCL-G004, SAM, June 1992.
Fal94: Kirsten Falkedal, editor. Proceedings of the of the Evaluators' Forum, 1991, Les Rasses, Vaud, Switzerland, April 1994. ISSCO, Geneva.
GSJ93: J. R. Galliers and Karen Sparck Jones. Evaluating natural language processing systems. Technical Report 291, University of Cambridge Computer Laboratory, March 1993. To appear in Springer Lecture Notes in Artificial Intelligence.
HAB91: P. Harrison, S. Abney, E. Black, D. Flickenger, C. Gdaniec, R. Grishman, D. Hindle, R. Ingria, M. Marcus, B. Santorini, and T. Strzalkowski. Evaluating syntax performance of parser/grammars of English. In Proceedings of the Workshop On Evaluating Natural Language Processing Systems. Association For Computational Linguistics, 1991.
Har93a: D. Harman. Overview of the first Text REtrieval Conference (TREC-1). In Harman [Har93b], pages 1--20.
Har93b: Donna Harman, editor. National Institute of Standards and Technology Special Publication No. 500-207 on the The First Text REtrieval Conference (TREC-1), Washington, DC, 1993. National Institute of Standards and Technology, U.S. Department of Commerce, U.S. Government Printing Office.
Har94: Donna Harman, editor. National Institute of Standards and Technology Special Publication No. 500-215 on the The Second Text REtrieval Conference (TREC-2), Washington, DC, 1994. National Institute of Standards and Technology, U.S. Department of Commerce, U.S. Government Printing Office.
Hau94: R. Hausser. The coordinator's final report on the first Morpholympics. LDV-Forum, 11(1):54--64, 1994.
HHM92: M. Höge, A. Hohmann, and R. Mayer. Evaluations of TWB: Operationalization and test results. Final Report of the ESPRIT I Project 2315 Translators' Workbench (TWB), 1992.
HHvdH93: M. Höge, A. Hohmann, K. van der Horst, S. Evans, and H. Caeyers. User participation in the TWB II project: The first test cycle. Report of the Esprit II Project 6005 Translators' Workbench II (TWB II), 1993.
HS84: T. Houtgast and H. J. M. Steeneken. A multi-lingual evaluation of the Rasti-method for estimating speech intelligibility in auditoria. Acustica, 54:185--199, 1984.
HS92: W. J. Hutchins and H. L. Somers. An introduction to machine translation. In An introduction to Machine Translation. Academic Press, London, 1992.
HWHK65: A. S. House, C. E. Williams, M. H. L. Hecker, and K. D. Kryter. Articulation testing methods: Consonantal differentiation with a closed-response set. Journal of the Acoustical Society of America, 37:158--166, 1965.
ICA84: Institute of Electrical and Electronic Engineers. Proceedings of the 1984 International Conference on Acoustics, Speech, and Signal Processing, 1984.
ICD93: AIPR-IEEE. Proceedings of the Second International Conference on Document Analysis and Recognition, Tsukuba Science City, Japan, October 1993. IAPR.
ICS94: Proceedings of the 1994 International Conference on Spoken Language Processing, Yokohama, Japan, September 1994.
Ish83: K. Ishii. Generation of distored charaters and its applications. System, Computer, Controls, 14(6):1270--1277, 1983.
ISO91: ISO. Information technology---software product evaluation, quality characteristics and guidelines for their use. Technical Report 9126, International Organization for Standardization, 1991.
ITU93: ITU. ITU-TTS draft recommendation p.8s: Subjective performance assessment of the quality of speech voice output devices. Technical Report COM 12-6-E, International Telecommunication Union, 1993.
Jek93: U. Jekosch. Speech quality assessment and evaluation. In Eurospeech [Eur93], pages 1387--1394. Keynote address.
JM92: K. Jones and J. Mariani, editors. Proceedings of the 1992 Workshop of the International Committee on Speech Databases and I/O Systems Assessment. COCOSDA, 1992.
KD91: D. Karis and K. M. Dobroth. Automating services with speech recognition over the public switched telephone network: Human factors considerations. IEEE Journal of Selected Areas in Communications, 9(4):574--585, May 1991.
KF90: M. King and K. Falkedal. Using test suites in evaluation of MT systems. In Proceedings of the 28th Annual Meeting of the Association for Computational Linguistics, volume 2, pages 211--216, Pittsburgh, Pennsylvania, 1990. Association for Computational Linguistics.
KHP93: T. Kanungo, R. M. Haralick, and I. Phillips. Global and local document degradation models. In ICDAR [ICD93], pages 730--736.
KKSF93: H. Klaus, H. Klix, J. Sotscheck, and K. Fellbaumn. An evaluation system for ascertaining the quality of synthetic speech based on subjective category rating tests. In Eurospeech [Eur93], pages 1679--1682.
KRNN93: J. Kanai, S. V. Rice, T. A. Nartker, and G. Nagy. Performance metrics for document understanding systems. In ICDAR [ICD93], pages 424--427.
Kry62: K. D. Kryter. Methods for the calculation and use of the articulation index. J. of the Acoustical Society of America, 34:1689--1697, 1962.
LB88: J. Lehrberger and L. Bourbeau. Machine translation: linguistic characteristics of MT systems and general methodology of evaluation. John Benjamins, Amsterdam, Philadelphia, 1988.
LGP89: J. S. Logan, B. G. Greene, and D. B. Pisoni. Segmental intelligibility of synthetic speech produced by rule. Journal of the Acoustical Society of America, 86(2):566--581, 1989.
LLT94: Y. Li, D. Lopresti, and A. Tomkins. Validation of document image defect models for optical character recognition. In Proceedings of the 3rd Annual Symposium on Document Analysis and Information Retrieval, pages 137--150, University of Nevada, Las Vegas, 1994.
MNY93: T. Matsui, T. Noumi, I. Yamashita, T. Watanabe, and M. Yoshimuro. State of the art of handwritten numeral recognition in Japan---the results of the first IPTP character recognition competition. In ICDAR [ICD93], pages 391--396.
Moo94: R. C. Moore. Semantic evaluation for spoken-language systems. In ARPA [ARP94].
Nag94: G. Nagy. Validation of simulated OCR data sets. In Proceedings of the 3rd Annual Symposium on Document Analysis and Information Retrieval, pages 127--135, University of Nevada, Las Vegas, 1994.
NI92: H. Nomura and H. Isahara. JEIDA's criteria on machine translation evaluation. In Proceedings of the International Symposium on Natural Language Understanding and AI, Kyushu Institute of Technology, Iizuka, Japan, 1992. part of the International Symposia on Information Sciences.
NND93: J. Nerbonne, K. Netter, A. K. Diagne, J. Klein, and L. Dickmann. A diagnostic tool for German syntax. Machine Translation, 8:85--107, 1993.
OCW94: S. L. Oviatt, P. R. Cohen, and M. Q. Wang. Toward interface design for human language technology: Modality and structure as determinants of linguistic complexity. Speech Communication, 15(3--4):283--300, December 1994.
Ovi95: S. L. Oviatt. Predicting spoken disfluencies during human-computer interaction. Computer Speech and Language, 9:19--35, 1995.
PFF94: D. Pallett, J. Fiscus, W. Fisher, J. Garofolo, B. Lund, and M. Prysbocki. 1993 benchmark tests for the ARPA spoken language program. In ARPA [ARP94], pages 49--74.
PJ94: Louis C. W. Pols and U. Jekosch. A structured way of looking at the performance of text-to-speech systems. In Proceedings, ESCA/IEEE Synthesis Workshop, pages 203--206, New Paltz, New York, 1994.
Pol91: Louis C. W. Pols. Quality assessment of text-to-speech synthesis-by-rule. In S. Furui and M. M. Sondhi, editors, Advances in speech signal processing, chapter 13, pages 387--416. Marcel Dekker, New York, 1991.
Pol94a: Louis C. W. Pols. Speech technology systems: Performance and evaluation. In R. E. Asher, editor, The Encyclopedia of Language and Linguistics, volume 8, pages 4289--4296. Pergamon Press, Oxford, 1994.
Pol94b: Louis C. W. Pols. Voice quality of synthetic speech: Representation and evaluation. In ICSLP [ICS94], pages 1443--1446.
PSp92: Louis C. W. Pols and SAM-partners. Multi-lingual synthesis evaluation methods. In Proceedings of the 1992 International Conference on Spoken Language Processing, volume 1, pages 181--184, Banff, Alberta, Canada, October 1992. University of Alberta.
Ric93: S. V. Rice. The OCR experimental environment, version 3. Technical Report ISRI TR-93-04, University of Nevada, Las Vegas, Nevada, 1993.
Rin93: A. Rinsche. Evaluationsverfahren für maschinelle übersetzunngssysteme: zur methodik und experimentellen praxis. Technical report, Kommission der Europaeischen Gemeinschaften, Bericht EUR 14766 DE, 1993.
RKN93: S. V. Rice, J. Kanai, and T. A. Nartker. An evaluation of OCR accuracy. Technical Report ISRI TR-93-01, University of Nevada, Las Vegas, Nevada, 1993.
RKN94: S. V. Rice, J. Kanai, and T. A. Nartker. The third annual test of OCR accuracy. Technical Report ISRI TR-94-03, University of Nevada, Las Vegas, Nevada, 1994.
RW93: J. R. Rhyne and C. G. Wolf. Recognition-based user interfaces. In H. R. Hartson and D. Hix, editors, Advances in Human-Computer Interaction, volume 4, chapter 7, pages 191--250. Ablex Publishing Corp, Norwood, New Jersey, 1993.
SH80: H. J. M. Steeneken and T. Houtgast. A physical method for measuring speech-transmission quality. J. Acoustical Society of America, 67:318--326, 1980.
SJ94: Karen Sparck Jones. Towards better NLP system evaluation. In ARPA [ARP94].
SK72: H. W. Sinaiko and G. R. Klare. Further experiments in language translation: readability of computer translations. ITL, 15:1--29, 1972.
SK73: H. W. Sinaiko and G. R. Klare. Further experiments in language translation: a second evaluation of the readability of computer translations. ITL, 19:29--52, 1973.
Sor94: C. Sorin. Towards high-quality multilingual text-to-speech. In Proceedings of the CRIM/FORWISS Workshop on Progress and Prospects of Speech Research and Technology, pages 53--62, Münich, 1994.
Spi91: J. Spitz. Collection and analysis of data from real users: Implications for speech recognition/understanding systems. In DARPA [DAR91b].
Spi93: M. F. Spiegel. Using the ORATOR synthesizer for a public reverse-directory service: Design, lessons, and recommendations. In Eurospeech [Eur93], pages 1897--1900.
Ste92: H. J. M. Steeneken. Quality evaluation of speech processing systems. In Nejat Ince, editor, Digital Speech Coding: Speech coding, Synthesis and Recognition, chapter 5, pages 127--160. Kluwer Norwell, USA, 1992.
SVH93: H. J. M. Steeneken, J. Verhave, and T. Houtgast. Objective assessment of speech communication systems; introduction of a software based procedure. In Eurospeech [Eur93], pages 203--206.
Tho92: H. Thompson, editor. The Strategic Role of Evaluation in Natural Language Processing and Speech Technology. Human Communication Research Centre, University of Edinburgh, 1992.
VS82: G. Van Slype. Conception d'une méthodologie générale d'évaluation de la traduction automatique. Multilingua, 1(4):221--237, 1982.
W94: J. S. White et al. The ARPA MT evaluation methodologies: evolution, lessons, and future approaches. In Technology partnerships for crossing the language barrier: Proceedings of the 1st Conference of the Association for Machine Translation in the Americas, pages 193--205, Washington, DC, October 1994. Association for Machine Translation in the Americas.
WB95: S. E. Wright and G. Budin, editors. Handbook of Terminology Management. John Benjamins, Amsterdam/Philadelphia, 1995. winter 1995.
WGJ92: R. A. Wilkinson, J. Geist, S. Janet, P. J. Grother, C. J. C. Burges, R. Creecy, B. Hammond, J. J. Hull, N. J. Larsen, T. P. Vogl, and C. L. Wilson. The first census optical character recognition systems conference. Technical Report NISTIR-4912, National Institute of Standards and Technology, U.S. Department of Commerce, September 1992.
ZF91: E. Zoltan-Ford. How to get people to say and type what computers can understand. International Journal of Man-Machine Studies, 34:527--547, 1991.