Subjective speech quality measurement repeatability: comparison of laboratory test results

Abstract

This article reports on a multi-lab subjective listening experiment aiming at inter-lab and intra-lab test results repeatability verification. An identical set of speech samples corresponding to contemporary networks has been tested by three independent labs deploying ITU-T P.835 methodology. The tests results have been compared regarding Pearson correlation, RMSE, RMSE* and numbers of opposite pair-wise comparisons. The results show the level of inter-lab and intra-lab repeatability in the case of identical test speech samples utilization and thus confirm the subjective tests are highly repeatable in case they follow recommendation requirements strictly. The tests also show differences in results in case subject expectations are set differently using a wider set of test speech samples (as presented in one of the labs).