CodaLab Competition - wbap/WM_Hackathon Wiki
The CodaLab competition is here
- The class name
agent/agent.pyclass must remain the same
- The class name
gym_game/envs/agent_env.pymust remain the same
model_namevariable passed must be called
preprocessor_namevariable (if used) must be called
- The following configuration filenames should not be changed
agent_av_pretrained.json: ensure that you specify your valid checkpoint path here, with the correct configuration
dm2s_env_*.json: this includes
m2s_env_*.json: this includes
Preparing the Submission
To prepare for submitting your code to CodaLab, ensure that you've met all the naming requirements to ensure that your model is properly evaluated. It is advisable to remove any unnecessary logs, files or other data that was produced during training, in order to reduce the overall size of your submission. For example, TensorBoard events generated during training can be significantly large in size, as can intermediate model checkpoints.
After cleaning up the submission folder, you can compress its contents into a ZIP file. Ensure that you are archiving the contents of the submission folder, rather than the folder itself. For example, on macOS/Linux, rather than compressing entire
WM_Hackathon using right-click option then 'Compress', instead change into the folder and select all the files/folders inside then right-click and 'Compress'. This avoids creating an additional parent directory when extracting the archive. Alternatively, see below for a command-line example:
cd WM_Hackathon/ zip -r ../submission.zip .
Head over to the 'Participate' section in the CodaLab competition, and then to 'Submit / View Results'. Clicking the 'Submit' button will prompt you to select the submission archive ZIP file. Depending on the size, it may take a few minutes to be submitted and CodaLab does not provide a user friendly visual indication of this.
The evaluation script will import your custom agent model, by importing the
Agent class, and load the checkpoint specified in
agent_av_pretrained.json. It will run multiple evaluation runs, one for each mode (color, position, shape) within each game type (Delayed Match to Sample and Match to Sample). The individual run score will be averaged over the 10 trials, and total score will be computed as well.
The evaluation script will run your submission within a Docker container using the same Docker image available in the repository. As such, please ensure that your code works within that environment to ensure that your submission is successful.