Task
Participants will collaborate in teams (4 - 5
people) to devise machine learning-driven
solutions aimed at converting noisy observations (referred to as lightcurves) into transmission
spectra.
The objective is to approximate the ground truth spectra as closely as achievable. The winning
team(s)
will be announced as the victor(s) of the hackathon and will receive a prize.
Evaluation
Your solution will be judged on 2 criteria: 1. Its performance on the final
heldout
datasets and 2. the interpretability of the model. A panel of judges comprising of experts in
Astrophysics and Machine Learning will access and score your solution according to how
interpretable
they are. To ensure the models are comparable among teams, participants will work on a fixed
metric(See Scoring Metrics tab)
that measures the
distance between ground truth and the prediction. The best overall model in both criteria will
be
declared winner of this hackathon.
Model Restrictions
There is no restriction on the models, algorithms or data preprocessing techniques, neither on
the
programming languages, environments or tools used for your implementation. You are also free to
use
data
augmentation techniques, pretrained models or any prior domain knowledge not included in the
provided
dataset. Finally, you are free to choose your own way of splitting the training data between
training
and validation sets and to use as many of the provided datapoints or features as you wish – or
can
handle
The ML task to be solved is a supervised learning one, and more particularly a multi-targets
regression
problem.
Features:
Features:
Each training datapoint consists of a set of 55 noisy light curves (one per wavelength, each
being a
timeseries of 300 timesteps) and a set of 6 additional stellar and planetary parameters. All
these
are
real numbers. For more details on what a light curve is and what we are modelling, go to the
Science
page.
Targets:
The goal is to predict a set of 55 real values (relative radii, one per wavelength) for any
given
datapoint (lightcurve).
Event Rules
1. Each team must have a unique alias.
2. Team names containing any form of offensive language, including but not limited to profanity,
derogatory terms, hate speech, or discriminatory language based on race, gender, religion,
sexual
orientation, or any other characteristic, are strictly prohibited. Any team found to be in
violation
of this rule will be disqualified from participation in the event, and further disciplinary
action
may be taken as deemed appropriate by the organisers.
3. The organisers will use the provided code, models and solutions only for the purposes of
checking
whether the contest rules (5) & (6) are not broken. The organisers will not use any result
without the authors’ permission.
4. Participants must not use TauREx 3 or similar atmospheric retrieval codes to retrieve the
test-set forward models. This would be considered test-set leakage and against the spirit of the
competition.
5. Participants must not have access to the heldout testset's ground truth before the
competition’s closing date. If they do, they will be disqualified.
6. For an entry to count as a winning entry, it must not rely heavily (as judged by the
organisers)
on hard-coded elements that are solely deemed to be due to test-set leakage.
7. The organisers reserve the right to interpret and enforce the above rules at their discretion
to
maintain a respectful and inclusive environment for all participants.
In the case of a draw
Should two or more top-ranked participants have the same final score, we will require the
participants to submit their algorithms. This is to check that no plagiarism has occured.
Reasons for disqualification
In case of plagiarism, forced test set leakage, failure to produce a description of the solution
when
requested or to conform with the multiple accounts or the team submissions policies, a
participant
will be disqualified and the next-in-rank participant will be considered in their place.
To evaluate a new solution (model), the participants are requested to upload its predictions on the
test dataset via the score calulcator page. See the Data Formats page for information on the upload
format. A score will then be automatically calculated for the solution.
The score reported is based on the average of the absolute error per target (i.e. on the relative
radii) across all test set examples \(i\) and all wavelengths \(j\) and is given by:
$$Score = 10^4 - \frac{\sum_{i \in Test} \sum_{j=1}^{55} w_{ij} 2 y_{ij} |\hat{y}_{ij} -
y_{ij}|}{\sum_{i \in Test} \sum_{j=1}^{55} {w_{ij}}}10^6,$$
where $$y_{ij}$$ is the true relative radius and $$\hat{y}_{ij}$$ the predicted relative radius
of the $$\(j\)-th$$ wavelength of the $$\(i\)-th$$ test set example. $$w_{ij}$$ is treated as
uniform in this hackathon, meaning every wavelengths and test examples are treated as equally
weighted.
The higher the score, the better your ranking. The maximum achievable score is 10000. The score is
not lower-bounded (i.e. can be negative), but reasonable attempts (e.g. predicting the average
target value for all test datapoints) should not produce scores below 4000.
https://colab.research.google.com/drive/18t-a781ZDPCg_E_mATPrawSkg9M7fOWn?usp=sharing