CodaLab Competition - wbap/WM_Hackathon Wiki

The CodaLab competition is here

Submission Process

Naming Restrictions

• The class name Agent and filepath agent/agent.py class must remain the same
• The class name AgentEnv and filepath gym_game/envs/agent_env.py must remain the same
• The model_name variable passed must be called agent_model
• The preprocessor_name variable (if used) must be called obs_preprocessor
• The following configuration filenames should not be changed
  • agent_av_pretrained.json: ensure that you specify your valid checkpoint path here, with the correct configuration
  • agent_env_av.json
  • dm2s_env_*.json: this includes color, position and shape variants
  • m2s_env_*.json: this includes color, position and shape variants

Preparing the Submission

To prepare for submitting your code to CodaLab, ensure that you've met all the naming requirements to ensure that your model is properly evaluated. It is advisable to remove any unnecessary logs, files or other data that was produced during training, in order to reduce the overall size of your submission. For example, TensorBoard events generated during training can be significantly large in size, as can intermediate model checkpoints.

After cleaning up the submission folder, you can compress its contents into a ZIP file. Ensure that you are archiving the contents of the submission folder, rather than the folder itself. For example, on macOS/Linux, rather than compressing entire WM_Hackathon using right-click option then 'Compress', instead change into the folder and select all the files/folders inside then right-click and 'Compress'. This avoids creating an additional parent directory when extracting the archive. Alternatively, see below for a command-line example:

cd WM_Hackathon/
zip -r ../submission.zip .

Head over to the 'Participate' section in the CodaLab competition, and then to 'Submit / View Results'. Clicking the 'Submit' button will prompt you to select the submission archive ZIP file. Depending on the size, it may take a few minutes to be submitted and CodaLab does not provide a user friendly visual indication of this.

Evaluation

The evaluation script will import your custom agent model, by importing the Agent class, and load the checkpoint specified in agent_av_pretrained.json. It will run multiple evaluation runs, one for each mode (color, position, shape) within each game type (Delayed Match to Sample and Match to Sample). The individual run score will be averaged over the 10 trials, and total score will be computed as well.

The evaluation script will run your submission within a Docker container using the same Docker image available in the repository. As such, please ensure that your code works within that environment to ensure that your submission is successful.