Transfer learning for Neural Network

Steps to do transfer learning in Shifu

Train ModelA as usual

• Run $ shifu stats
• Run $ shifu norm
• Run $ shifu varsel
• Run $ shifu train

Copy ModelA to ModelB

• Change 'basic.name' to ModelB in ModelConfig.json
• See: How to do it in PyShifu

Change ModelB model structure

• First of all, change train.isContinuous to true in ModelConfig.json
• Design the model structure - the ModelB's model structure should be larger than ModelA's. See:

The larger means wider , or deeper, or both wider and deeper.

Fix layers that in older model structrue

• Add FixedLayers in train.params, it is a list of layers that you want to freeze Please note, the layers start from 1, 2..., the parameters between input layer and first hidden layer, is defined as layer 1, the parameters between first hidden layer and second hidden layer is defined as layer 2... Example "FixedLayers" : [1, 2]. See:

• Add FixedBias in train.params, if you want to freeze the bias also. Example "FixedBias" : true

Train ModelB as usual

Example of setting

    "baggingNum" : 1,
    ...
    "numTrainEpochs" : 200,
    "isContinuous" : true,
    "workerThreadCount" : 4,
    "algorithm" : "NN",
    "params" : { 
      "Propagation" : "R",
      "LearningRate" : 0.1,
      "NumHiddenNodes" : [ 50, 10],
      "NumHiddenLayers" : 2,
      "RegularizedConstant" : 0.0,
      "ActivationFunc" : [ "tanh", "tanh" ],
      "FixedLayers" : [ 1 ],
      "FixedBias" : false 
    },