Zerospeech 2015

The 2015 Challenge appeared as a special session at Interspeech 2015 (Sept, 6-10, 2015, Dresden, see the Interspeech 2015 proceedings. The challenge’s aims were presented in [1] and the main results summarized in [2]. The references for Track 1 are in [3], [4], [5], [6], [7], [8] and for Track 2 in [9], [10]. Further papers were published in the SLTU 2016 special topic on Zero resource speech technology and elsewhere [11], [12], [13].

Track 1

Baseline and topline

The baseline and topline ABX error rates for Track 1 are given in Table 1 (see also [1]). For the baseline model, we used 13 dimensions MFCC features computed every 10ms and the ABX score was computed using the cosine distance. For the topline model, we used posteriorgrams extracted from a Kaldi GMM-HMM pipeline with MFCC and Delta and Delta-Delta features, Gaussian mixtures, triphone word-position-dependent states, fMLLR talker adaptation, with a bigram word language model. The exact same Kaldi pipeline was used for the two languages and gave a phone error rate (PER) of 26.4% for English, and 7.5% for Tsonga. Note that the two corpora are quite different: The English corpus contains spontaneous, casual speech; the Tsonga corpus contain read speech constructed out of a small vocabulary, and tailored for constructing speech recognition applications. The acoustic and language models were trained on the part of the corpora not used in the evaluation, and the posterior fed into the ABX evaluation software using the KL divergence. Unsupervised models are expected to fall in between the performance of these two systems.

System’s comparisons

Table 1. Track 1 - ABX across and within speaker results, on English and Xitsonga Datasets

	Authors	Ref.	Team	With Supervision	Binary	English		Xitsonga
						across	within	across	within

If your result does not appear there, please email us.

Track 2

Baseline and topline

For the baseline model, we used the JHU system described in Jansen & van Durme (2011) on PLP features. It performs DTW matching and uses random projections for increasing efficiency, and uses connected component clustering as a second step. The topline is an Adaptor Grammar using a unigram grammar, run on the gold phoneme transcription. Here, the topline performance is probably not attainable by unsupervised systems since it uses the gold transcription. It is more of a reference for the maximum value that it reasonable to expect for the metrics used.

System’s comparisons

Table 2. Track 2 metrics on English and Xitsonga datasets.

	Authors	Team	English						Xitsonga
			NED	cov	Boundary F-score	Token F-score	Type F-score	grouping F-score	NED	cov	Boundary F-score	Token F-score	Type F-score	grouping F-score

Challenge References