Image Classifier - Simsso/NIPS-2018-Adversarial-Vision-Challenge GitHub Wiki
Overview about our approaches of tackling the image classification task on Tiny ImageNet.
Models
This table contains the test accuracy of noteworthy training runs.
| Name | Test acc. [%] | Steps | Notes |
|---|---|---|---|
| resnet003 | ? | Configuration from here | |
| resnet_t01 | 49.11 | 131.3k | ResNet baseline (re-written code) |
| resnet_t34a_adam | 53.53 | 48.4k | Deeper ResNet with Adam optimizer |
| resnet_crowdai_dr0.8 | 55.81 | 60.92k | Aggressive dropout |
[1] Measurement was only taken on parts of the validation dataset (1k batch)
Adversarial Logit Pairing
Code at TF models GitHub repo contains checkpoints and training procedures for Tiny ImageNet classifiers. The paper can be found here.
It seems to be a reasonable approach to load the model weights and fine-tune.
Shell commands
The code is not Python 3.6 compatible; Python 2.7 works.
Training from scratch
python train.py --model_name="resnet_v2_50" --hparams="train_adv_method=clean" --dataset="tiny_imagenet" --dataset_image_size=64 --output_dir="./tmp/adv_train" --tiny_imagenet_data_dir="tiny-imagenet-tfrecord"
Evaluation
python eval.py --train_dir=./tmp/adv_train --trainable_scopes="resnet_v2_50/logits,resnet_v2_50/postnorm" --dataset=tiny_imagenet --dataset_image_size=64 --adv_method=clean --hparams="eval_batch_size=50" --eval_once=True --tiny_imagenet_data_dir="tiny-imagenet-tfrecord"
Related Work
- Stanford student paper (pre-trained Inception v3 on Tiny ImageNet)
ResNet with Trainable VQ-Layers
We have conducted several experiments with trainable vector quantization (VQ) layers in the ResNet with ALP weights. Trainable VQ-layers learn their embedding space with gradient descent on the loss surface induced by three components: alpha, beta, and coulomb loss.
The three losses have the following effects (intuitively):
- alpha: moves embedding space vectors closer to the inputs that were projected onto them
- beta: moves all embedding space vectors towards the mass of inputs
- coulomb: discourages embedding space vectors to be close to each other
The three hyperparameters were tuned with the goal of increasing the accuracy on validation data and increasing the minimal distance between each embedding vector and its closest neighbor (supposedly equivalent to the robustness of the model). Both measures (accuracy scalar and min distance histogram) were logged to TensorBoard. Eventually, hyperparameters with the magnitudes (rough values) alpha=1e-1, beta=2e-4, gamma=5e-4 seemed to be appropriate. These values were found by looking at the effect that each part of the loss has on the total loss term. That way alpha could be set to have a share of ~50%, beta 25%, and gamma 25%.
Some noteworthy submissions are (resnet-base-newcoulxn128x5em4x2repl, resnet-base-parallel3x16x64x128).