Thursday, September 22, 18:00 - 20:00

Sebastian Möller (Deutsche Telekom Laboratories, TU Berlin)

Evaluation of Spoken Dialogue

Abstract:
Despite the progress that speech transmission and speech and language technology has made over the past years, the quality of speech-based communication - be it between humans, be it between human and computer - often does not reach a level which is acceptable for the human communication partner. Several reasons are responsible for this fact: Human conversations are largely affected by increasing degradations of the transmission channel (telephone or packet-based network), e.g. by packet loss or delay. Furthermore, human-computer interaction depends on the performance of speech recognition and understanding modules, the dialogue manager, as well as on text-to-speech systems used for speech output. In order to design systems which are finally accepted by their (human) users, a profound knowledge of quality perception and assessment is necessary. Ideally, it would be desirable to predict communication quality on the basis of measurable characteristics of all technical systems involved in the communication process.

It is the aim of this lecture to present new approaches for reaching this target. Starting from a definition of quality which takes the human perception as the reference, two communication scenarios are addressed in more detail: The human-to-human communication over an impaired telephone channel and the interaction with a spoken dialogue system. For both scenarios, taxonomies are developed which describe different aspects of quality experienced by the user, as well as the (technical) factors which are in the hands of the system developer for optimizing these quality aspects. Assessment and evaluation methods are presented for quantifying several quality aspects, as well as algorithms which predict quality - as it is perceived by the human communication partner - on the basis of signals or parameters describing the involved technical systems. The prediction accuracy and the limitations of such algorithms are discussed, and necessary research is identified in order to better predict quality for future wideband and multimodal communication situations.