Test List - psfoley/Challenge GitHub Wiki

A starting point for tests that need to be run:

1. Go through the existing jupyter notebook and make a note of any issues during setup
2. Perform multiple experiment runs without resetting kernel
3. Ensure loss is not passed to the custom aggregation function Fixed
4. Make changes to the following functions that are intended to be configured by participants to see what breaks:
   • aggregation_function
   • validation_functions
     1. Create two validation functions, but give their labels the same name [(‘acc’: accuracy),(‘acc’: f1score)].
   • choose_training_collaborators
     1. Attempt to select collaborator for training that is not part of the experiment
   • training_hyper_parameters_for_round
     1. Return hyper parameters that are not supported (something besides epochs, learning_rate, etc.