Finetuning Pretrained Model with Embedding of Domain and Language Information for ASR of Very Low-Resource Settings

Abstract

This study investigates the effective incorporation of meta-information such as domain and language in finetuning a pretrained model based on self-supervised learning (SSL) for automatic speech recognition (ASR) in very low-resource settings. SSL pretrained models have been shown to achieve comparable or even better performance to conventional end-to-end systems even when we finetune them with a small dataset. However, it still requires the specific target dataset with a considerable amount of labeled data, like 10 hours, to achieve satisfactory performance. Thus, we propose to exploit heterogeneous datasets which are partially matched either in language or domain and apply multi-task learning (MTL) or adversarial learning using the meta-information. The finetuning comprises (1) domain adaptation, which uses in-domain multi-lingual datasets, and (2) language adaptation, which uses datasets of the same language but different domains. The auxiliary task is domain identification for language adaptation and language identification for domain adaptation. We then embed the output of the auxiliary task into the encoder output of the ASR task. The target dataset is the Khmer corpus of ECCC (the Extraordinary Chambers in the Courts of Cambodia) in various sizes from one hour to 10 hours. The experimental evaluations demonstrate that fusing the meta-information in MTL or adversarial learning significantly improves ASR accuracy. Moreover, a two-step adaptation method which first conducts domain adaptation and then language adaptation is the most effective. We also show that the target labeled dataset of only 5 hours gives an almost saturated performance.