Documentation

Scoring System

Light Track

Submissions will be evaluated by comparing distance between the quartiles estimates of each target given by Nested Sampling () and the submission (), where (stands for 16th, 50th and 84th percentile). For each test case we will compute the average relative RMSE out of all quartiles and targets, i.e.

The participants’ performance on the Light Track will be measured based on the average performance over the entire test set, i.e. . The scoring function function is computed in the following way:


Thus the higher the score, the better the performance. Winners will be selected based on the overall score of the final evaluation test set.

To help you with the model evaluation, we have included the exact metric we used for the challenge here.

Regular Track

We use the Wasserstein-2 distance, also known as Earth-Mover distance. It is a metric originating from the theory of optimal transport and is used to evaluate the level of overlap between two multivariate distributions, with 0 indicating maximally similar (i.e. the same) distributions and 1 maximally dissimilar.

where and represent the Nested Sampling (NS) generated approximate conditional distribution and the participant’s surrogate distribution for a single test case (n) respectively. represents the set of probability distributions on the metric space , whose marginal distributions are and on first and second factor respectively.
The overall score will be the average score obtained over the entire uploaded test set,

We subtract by unity and multiply the result by 1000 to turn it into a monotonically increasing function

To help you with the model evaluation, we have included the exact metric we used for the challenge here.