Loss [Calculate, Function] - Paxoo/PyTorch-Best_Practices GitHub Wiki

Overview

There a two used ways to calculate the loss:

Calculate loss average of the batch

The current loss average of the batch will be added to running_loss, printed after a number of mini-batches and reset. (number of mini-batches is set to 2000)

for epoch in range(..):
    running_loss = 0
    for batch in train_data:
        ...
        loss.backward()
        optimizer.step()
        running_loss += loss.item()

        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                        (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

Calculate loss for each epoch

It's a bit more accurate then the first one. First we multiply loss.item() with the batch size since loss.item() returns the average loss for each sample within the batch, so to get per sample overall we have to correct for that. Depending on the batch size and the length of your dataset, the last batch might be smaller than the specified batch size. If that’s the case, you multiply the loss by the current batch size inputs.size(0). Then we divide by the total length of the dataset/the amount of used mini-batches.

for epoch in range(..):
    running_loss = 0

    for inputs, labels in train_data:
        ...
        loss.backward()
        optimizer.step()
        running_loss += loss.item() * inputs.size(0)

    epoch_loss = running_loss / len(train_data)
    print(epoch_loss)

Custom Loss Function

PyTorch comes with many standard loss functions available for you to use in the torch.nn module: PyTorch Loss Functions PyTorch makes it very easy to extend this and write your own custom loss function. Here is a dummy implementation of Cross Entropy Loss:

def myCrossEntropyLoss(outputs, labels):
    batch_size = outputs.size()[0]            # batch_size
    outputs = F.log_softmax(outputs, dim=1)   # compute the log of softmax values
    outputs = outputs[range(batch_size), labels] # pick the values corresponding to the labels
    return -torch.sum(outputs)/num_examples

...
output = model(input)
loss = myCrossEntropyLoss(outputs, labels)`
loss.backward()
...