CodaLab Competition - wbap/WM_Hackathon GitHub Wiki
The CodaLab competition is here
Submission Process
Naming Restrictions
- The class name
Agent
and filepathagent/agent.py
class must remain the same - The class name
AgentEnv
and filepathgym_game/envs/agent_env.py
must remain the same - The
model_name
variable passed must be calledagent_model
- The
preprocessor_name
variable (if used) must be calledobs_preprocessor
- The following configuration filenames should not be changed
agent_av_pretrained.json
: ensure that you specify your valid checkpoint path here, with the correct configurationagent_env_av.json
dm2s_env_*.json
: this includescolor
,position
andshape
variantsm2s_env_*.json
: this includescolor
,position
andshape
variants
Preparing the Submission
To prepare for submitting your code to CodaLab, ensure that you've met all the naming requirements to ensure that your model is properly evaluated. It is advisable to remove any unnecessary logs, files or other data that was produced during training, in order to reduce the overall size of your submission. For example, TensorBoard events generated during training can be significantly large in size, as can intermediate model checkpoints.
After cleaning up the submission folder, you can compress its contents into a ZIP file. Ensure that you are archiving the contents of the submission folder, rather than the folder itself. For example, on macOS/Linux, rather than compressing entire WM_Hackathon
using right-click option then 'Compress', instead change into the folder and select all the files/folders inside then right-click and 'Compress'. This avoids creating an additional parent directory when extracting the archive. Alternatively, see below for a command-line example:
cd WM_Hackathon/
zip -r ../submission.zip .
Head over to the 'Participate' section in the CodaLab competition, and then to 'Submit / View Results'. Clicking the 'Submit' button will prompt you to select the submission archive ZIP file. Depending on the size, it may take a few minutes to be submitted and CodaLab does not provide a user friendly visual indication of this.
Evaluation
The evaluation script will import your custom agent model, by importing the Agent
class, and load the checkpoint specified in agent_av_pretrained.json
. It will run multiple evaluation runs, one for each mode (color, position, shape) within each game type (Delayed Match to Sample and Match to Sample). The individual run score will be averaged over the 10 trials, and total score will be computed as well.
The evaluation script will run your submission within a Docker container using the same Docker image available in the repository. As such, please ensure that your code works within that environment to ensure that your submission is successful.