How to participate

Choosing a train dataset

You can train on any of the standard ZeroSpeech Task 1 train sets listed on the Benchmarks and Datasets page, together or combined. You can also train on external datasets, as long as they are publicly available. During the submission process, you will be asked to specify what dataset was used to train your system, providing a link (or publication reference) if it is an external dataset.

The provided datasets can be downloaded using our toolkit or directly using the provided URLs in our repository.

Setting up the zrc toolkit

You will need to use the zrc toolkit to manage, evaluate, and upload your submissions. Please see the Getting Started section of the website for more information.

Creating your submission

You must create one submission for each benchmark.

To initialize your submission, use the toolkit command

zrc submission:init <name> <location>

to create an empty submission folder, where <name> is the name of the benchmark (abx15, abx17, abxLS) and location is the path where the directory will be created.

Each of the following sections describes the contents that must be in the submission before it can be evaluated.

meta.yaml

This file contains meta information about the author and how this submission was created.

example:

model_info:
  model_id: null
  gpu_budget: 60
  system_description: "CPC-big (trained on librispeech 960), kmeans (trained on librispeech 100), LSTM. See https://zerospeech.com/2021 for more details."
  train_set: "librispeech 960, librispeech 100"
publication:
  author_label: "Nguyen et al."
  authors: "Nguyen, T., Seyssel, M., Rozé, P., Rivière, M., Kharitonov, E., Baevski, A., Dunbar, E. & Dupoux, E."
  paper_title: "The zero resource speech benchmark 2021: Metrics and baselines for unsupervised spoken language modeling."
  paper_url: "https://arxiv.org/abs/2011.11589"
  publication_year: 2021
  institution: "EHESS, ENS, PSL Research University, CNRS and Inria"
  team: "CoML Team"
code_url: "https://github.com/zerospeech/zerospeech2021_baseline"
open_source: true

We encourage all participants to make their code open source and to provide a working, permanent link to it.

params.yaml

This file contains parameters that can override the defaults of each benchmark.

cuda: <bool> specifies whether GPU acceleration is enabled for the evaluation
distance_mode: <str> The metric to use for the abx distance d must be one of the  following values
 'euclidean', 'cosine', 'kl' or 'kl_symmetric'. The default used will be 'cosine'
 **WARNING** the 'cosine' metric here refers to an angular distance as in the usual ABX evaluation.
feature_size: <float> Shift (in s) between two features frames

model outputs

For the abx benchmarks each file in the dataset must have an associated 2D array, stored in NumPy format.

• These need to be organized in a similar fashion as in the original dataset.
• All numbers in the array are encoded as floats.
• The number of rows (the time dimension) depends on the duration of the individual speech sample
• The number of columns (the feature dimension) must be constant across all the files.
• The frame shift (the shift between two successive frames) must be given in params.yaml along with the metric used for evaluation of those features.
• Each array must contain at least 2 frames (i.e. at least two rows)

Structure of files for each ABX benchmark :

• abxLS

dev-clean/*.npy
dev-other/*.npy
test-clean/*.npy
test-other/*.npy

• abx17

english/
  1s/*.npy
  10s/*.npy
  120s/*.npy
french/
  ...
mandarin/
  ...
german/
  ...
wolof/
  ...

• abx15

embeddings/*.npy

Running the evaluation

Once the submission has been successfully created we can now run the evaluation. Depending on your benchmark choice you can use the following commands to run the evaluation:

• zrc benchmarks:run abx15 /path/to/submission
• zrc benchmarks:run abx17 /path/to/submission
• zrc benchmarks:run abxLS /path/to/submission

Your results are created in the /path/to/submission/scores directory.

Notes:

• A validation will run before each evaluation, to skip use the option --skip-validation
• If the dataset has subsets you can run the eval on only a selected subset, e.g., --sets dev, --sets english, etc.
• Sub-subsets are specified with the --task parameter, e.g., --task clean

You can see the list of the available benchmarks by using zrc benchmarks

Uploading Results

To submit your results to the public leaderboard, you must create an account on our website. Follow this link to create your account.

Then, using the toolkit, create a local session:

zrc user:login

providing your username & password.

Once this is done, you can upload using the following command:

zrc submit <submission_dir>

Multiple Submissions

If your system can be used for multiple tasks (for example, Task 1 and Task 3, Task 1 and Task 4), you are strongly encouraged to make submission to all the tasks you can. To link submissions of a single system across benchmarks, you need to use the same model_id in your meta.yaml auto-generated after the first submission.