How to participate
Choosing a train dataset
You can train on any of the standard ZeroSpeech Task 1 train sets listed on the Benchmarks and Datasets page, together or combined. You can also train on external datasets, as long as they are publicly available. During the submission process, you will be asked to specify what dataset was used to train your system, providing a link (or publication reference) if it is an external dataset.
Setting up the zrc toolkit
You will need to use the zrc toolkit to manage, evaluate, and upload your submissions. Please see the Getting Started section of the website for more information.
Creating your submission
You must create one submission for each benchmark.
To initialize your submission, use the toolkit command
zrc submission:init <name> <location>
to create an empty submission folder, where
<name> is the name of the benchmark (abx15, abx17, abxLS) and location is the path where the directory will be created.
Each of the following sections describes the contents that must be in the submission before it can be evaluated.
This file contains meta information about the author and how this submission was created.
model_info: model_id: null gpu_budget: 60 system_description: "CPC-big (trained on librispeech 960), kmeans (trained on librispeech 100), LSTM. See https://zerospeech.com/2021 for more details." train_set: "librispeech 960, librispeech 100" publication: author_label: "Nguyen et al." authors: "Nguyen, T., Seyssel, M., Rozé, P., Rivière, M., Kharitonov, E., Baevski, A., Dunbar, E. & Dupoux, E." paper_title: "The zero resource speech benchmark 2021: Metrics and baselines for unsupervised spoken language modeling." paper_url: "https://arxiv.org/abs/2011.11589" publication_year: 2021 institution: "EHESS, ENS, PSL Research University, CNRS and Inria" team: "CoML Team" code_url: "https://github.com/zerospeech/zerospeech2021_baseline" open_source: true
We encourage all participants to make their code open source and to provide a working, permanent link to it.
model_idparameter is generated when you submit a system to our backend. If you wish to submit the same system to multiple benchmarks, keep the model_id the same to allow our system to link the submissions.
This file contains parameters that can override the defaults of each benchmark.
cuda: <bool> specifies whether GPU acceleration is enabled for the evaluation distance_mode: <str> The metric to use for the abx distance d must be one of the following values 'euclidean', 'cosine', 'kl' or 'kl_symmetric'. The default used will be 'cosine' **WARNING** the 'cosine' metric here refers to an angular distance as in the usual ABX evaluation. feature_size: <float> Shift (in s) between two features frames
For the abx benchmarks each file in the dataset must have an associated 2D array, stored in NumPy format.
- These need to be organized in a similar fashion as in the original dataset.
- All numbers in the array are encoded as floats.
- The number of rows (the time dimension) depends on the duration of the individual speech sample
- The number of columns (the feature dimension) must be constant across all the files.
- The frame shift (the shift between two successive frames) must be given in
params.yamlalong with the metric used for evaluation of those features.
- Each array must contain at least 2 frames (i.e. at least two rows)
Structure of files for each ABX benchmark :
dev-clean/*.npy dev-other/*.npy test-clean/*.npy test-other/*.npy
english/ 1s/*.npy 10s/*.npy 120s/*.npy french/ ... mandarin/ ... german/ ... wolof/ ...
Running the evaluation
Once the submission has been successfully created we can now run the evaluation. Depending on your benchmark choice you can use the following commands to run the evaluation:
zrc benchmarks:run abx15 /path/to/submission
zrc benchmarks:run abx17 /path/to/submission
zrc benchmarks:run abxLS /path/to/submission
Your results are created in the
- A validation will run before each evaluation, to skip use the option
- If the dataset has subsets you can run the eval on only a selected subset, e.g.,
--sets english, etc.
- Sub-subsets are specified with the
You can see the list of the available benchmarks by using
To submit your results to the public leaderboard, you must create an account on our website. Follow this link to create your account.
Then, using the toolkit, create a local session:
providing your username & password.
Once this is done, you can upload using the following command:
zrc submit <submission_dir>
To submit your scores you need include all the required files in the same directory.
- source files: (embeddings/probabilities) these are files extracted from your model.
- score files: these are the result of the evaluation process.
- params.yaml: these are the parameters of the evaluation process.
- meta.yaml: generic information on submission
If your system can be used for multiple tasks (for example, Task 1 and Task 3, Task 1 and Task 4), you are strongly encouraged to make submission to all the tasks you can.
To link submissions of a single system across benchmarks, you need to use the same
model_id in your
meta.yaml auto-generated after the first submission.