Open Profiling of Quality: A Mixed Methods Research Approach for Audiovisual Quality Evaluations
Supervisor(s) and Committee member(s): Prof. Dr.-Ing. Karlheinz Brandenburg, Prof. Dr. Søren Bech, Prof. Dr. Patrick Le Callet
To meet the requirements of consumers and to provide them with a greater quality of experience than existing systems do is a key issue for the success of modern multimedia systems. However, the question about an optimized quality of experience becomes more and more complex as technological systems are evolving and several systems are merged into new ones, e.g. systems for mobile 3D television and video. To be able to optimize critical components of a system under development with as little perceptual errors as possible, user studies are conducted throughout the whole process. A variety of research methods for different purposes have been provided by standardization bodies since the 1970s. These methods allow researchers to evaluate the hedonic excellence of a set of test stimuli. However, a broader view to quality has been taken recently to be able to evaluate quality beyond its hedonic excellence to obtain a greater knowledge about perceived quality and its subjective quality factors that impact on the user.
The goal of this thesis is twofold. The primary goal is the development of a validated mixed- methods research approach for audiovisual quality evaluations. The method shall allow collecting quantitative and descriptive data during the experiment to combine evaluation of hedonic excellence and the elicitation of underlying subjective quality factors. The second goal is the application of the developed method within a series of studies in the domain of mobile 3D video and television to show its applicability.
Open Profiling of Quality (OPQ) is a mixed-methods research approach which combines a quan- titative, psychoperceptual evaluation of hedonic excellence and a descriptive sensory analysis of underlying quality factors based on naïve participants’ individual vocabulary. This combination allows defining the excellence of overall quality, understanding the characteristics of quality per- ception, and, eventually, constructing a link between preferences and quality attributes. The method was developed under constructive research with respect to validity and reliability of test results. A series of quality evaluation studies with more than 300 test participants was conducted along dif- ferent critical components of a system for optimized mobile 3DTV content delivery over DVB-H. The results complemented each other, and, even more importantly, quantitative quality preferences were explained by sensory descriptions in all studies.
Beyond the development of OPQ, the thesis proposes further research approaches, e.g. a con- ventional profiling in which OPQ’s individual vacobulary is substituted by a fixed set of Quality of Experience components or Descriptive Sorted Napping which combines a sorting task and a short post-task interview. All approaches are compared to Open Profiling of Quality at the end of the thesis. To be able to holistically contrast strengths and weaknesses of each method, a comparison model for audiovisual evaluation methods was developed and a first conceptual operationalization of the model was applied in the comparison.