This paper was accepted on the workshop “I Can’t Consider It’s Not Higher: Understanding Deep Studying By way of Empirical Falsification”
Steady pseudo-labeling (PL) algorithms comparable to slimIPL have just lately emerged as a robust technique for semi-supervised studying in speech recognition. In distinction with earlier methods that alternated between coaching a mannequin and producing pseudo-labels (PLs) with it, right here PLs are generated in end-to-end method as coaching proceeds, enhancing coaching velocity and the accuracy of the ultimate mannequin. PL shares a standard theme with teacher-student fashions comparable to distillation in {that a} trainer mannequin generates targets that must be mimicked by the coed mannequin being skilled. Nevertheless, apparently, PL methods typically use hard-labels, whereas distillation makes use of the distribution over labels because the goal to imitate. Impressed by distillation we anticipate that specifying the entire distribution (aka soft-labels) over sequences because the goal for unlabeled knowledge, as a substitute of a single finest go pseudo-labeled transcript (hard-labels) ought to enhance PL efficiency and convergence. Surprisingly and unexpectedly, we discover that soft-labels targets can result in coaching divergence, with the mannequin collapsing to a degenerate token distribution per body. We hypothesize that the rationale this doesn’t occur with hard-labels is that coaching loss on hard-labels imposes sequence-level consistency that retains the mannequin from collapsing to the degenerate answer. On this paper, we present a number of experiments that help this speculation, and experiment with a number of regularization approaches that may ameliorate the degenerate collapse when utilizing soft-labels. These approaches can convey the accuracy of soft-labels nearer to that of hard-labels, and whereas they’re unable to outperform them but, they function a helpful framework for additional enhancements.
