CROSS ENTROPY - beyondnlp/nlp GitHub Wiki

• torch.exp()를 사용하는 이유
  • focal_loss에 보면 torch.exp()를 사용하는 경우가 있다. 지수함수는 아래와 같이 모든 y값이 0 이상이다. 따라서 음수와 양수가 있는 값을 모두 양수로 치환할때 torch.exp()를 적용해 준다.
  • https://velog.io/@heaseo/Focalloss-%EC%84%A4%EB%AA%85

import random
import torch
import torch.nn as nn
import numpy as np


answer=[1,5]
prob=[]
for i in range( 2 ):
    plist=[]
    for j in range(10):
        rand = random.random()
        plist.append( rand )
    prob.append(plist)


def cross_entropy1( prob, answer ):
    output = torch.Tensor( prob )
    target = torch.LongTensor(answer)
    criterion = nn.CrossEntropyLoss()
    loss = criterion(output, target)
    return loss



def cross_entropy2( prob, answer ):
    loss=0
    for i in range( len(answer) ):
        target = answer[i]
        output = prob[i]
        idx = target
        loss += np.log(sum(np.exp(output))) - output[idx]
    loss = loss/len(answer)
    return loss

def cross_entropy3(y, t):

    y = np.array(y)
    t = np.array(t)
    if y.ndim == 1:
        t = t.reshape(1, t.size)
        y = y.reshape(1, y.size)

    # 훈련 데이터가 원-핫 벡터라면 정답 레이블의 인덱스로 반환
    if t.size == y.size:
        t = t.argmax(axis=1)

    batch_size = y.shape[0]

  
    return -np.sum(np.log(y[np.arange(batch_size), t] + 1e-7)) / batch_size

loss1 = cross_entropy1( prob, answer );
loss2 = cross_entropy2( prob, answer );
loss3 = cross_entropy3( prob, answer );
print(f"loss1 : {loss1}")
print(f"loss2 : {loss2}")
print(f"loss3 : {loss2}")

• hf compute_loss :

  • https://github.com/huggingface/transformers/blob/984bc11b0882ff1e5b34ba717ea357e069ceced9/src/transformers/trainer.py#L3363
  • https://github.com/huggingface/transformers/blob/984bc11b0882ff1e5b34ba717ea357e069ceced9/src/transformers/trainer_pt_utils.py#L545

• numpy array slicing idx1=[0, 1] idx2=[3, 5]
  prob[ idx1, idx2] => prob[0,3] => prob[1,5]

• 골빈해커의 3분 딥러닝 ( 63 page 참고 )

정답(Y)이 아래와 같고
[ 1 0 0 ], [ 0 0 1 ](/beyondnlp/nlp/wiki/-1-0-0-],-[-0-0-1-)
>>> Y=[1,0,0],[0,0,1](/beyondnlp/nlp/wiki/1,0,0],[0,0,1)

모델(Model)에서 나온 값이
[ 0.7 0.2 0.1 ],[0.2 0.3 0.5 ](/beyondnlp/nlp/wiki/-0.7-0.2-0.1-],[0.2-0.3-0.5-)처럼 나오면
>> model=[0.7,0.2,0.1],[0.2,0.3.0.5]]
이 두 값을 이용하여  CROSS ENTROPY를 계산할수 있다.

정답에 모델의 결과에 tf.log(model)취한 값을 곱한다.
>>> log_model=tf.log(model)
>>> sess=tf.Session()
>>> sess.run(output)
array([[-0.35667497, -1.609438  , -2.3025851 ],
       [-1.609438  , -1.2039728 , -0.6931472 ]], dtype=float32)


Y * tf.log(Model)
이제는 각 행별로 더하면 된다( reduce_sum(axis=1) )
>>> output=Y*log_model
>>> sess.run(output)
array([[-0.35667497, -0.        , -0.        ],
       [-0.        , -0.        , -0.6931472 ]], dtype=float32)

>>> a=tf.reduce_mean(output,1)
>>> sess.run(a)

array([-0.11889166, -0.23104906], dtype=float32)