[Word2Vec] Neural Language Model and Word2Vec - dsindex/blog GitHub Wiki

1. Word2Vec

• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• word2vec basic concept
• word2vec Parameter Learning Explained
• derivatives

# softmax, loglikelihood에서 주로 사용되는 수식

y = exp(x) / sum( exp(x') )

log( y ) = log( exp(x) ) - log( sum( exp(x') ) )
         = x - log( sum( exp(x') ) ) 
         = -E

# nagative sampling에서 주로 사용되는 수식

sig(x) = 1 /  (1+exp(-x) ) 
       = 1 / ( 1 + ( 1/exp(x) ) ) 
       = 1 / ( exp(x) + 1 ) / exp(x) 
       = exp(x) / ( 1+exp(x) )

1 - sig(x) = 1 - ( 1 / ( 1+exp(-x) ) ) 
           = ( ( 1 + exp(-x) ) - 1 ) / ( 1+exp(-x) )
           = exp(-x) / ( 1+exp(-x) ) 
           = ( 1 / exp(x) ) / ( 1 + ( 1/exp(x) ) )
           = ( 1 / exp(x) ) / ( exp(x) + 1 ) / exp(x) 
           = 1 / ( 1 + exp(x) )

sig(x)' = exp(x) / ( exp(x) + 1 )^2
        = ( exp(x) / ( 1+exp(x) ) ) * ( 1 / ( 1+exp(x) ) )
        = sig(x) * ( 1 - sig(x) )


log( sig(x) ) = log( 1 / (1+exp(-x) )

이걸 미분하면, 

( 1 / sig(x) ) * sig'(x) = ( 1 / sig(x) ) * sig(x) * ( 1 - sig(x) ) 
                         = 1 - sig(x)

즉, log( sig(x) )' = 1 - sig(x)


log( sig(-x) ) 

이걸 미분하면

( 1 / sig(-x) ) * sig'(-x) * (-1) 
   = ( 1 / sig(-x) ) * sig(-x) * ( 1 - sig(-x) ) * (-1) 
   = - 1 + sig(-x) 
   = -( 1 - ( 1 / ( 1+exp(x) ) ) ) 
   = -( ( 1+exp(x) ) - ( 1  / ( 1+exp(x) ) 
   = -( exp(x) / ( 1+exp(x) ) )
   = -sig(x)

즉,  log( sig(-x) )' = -sig(x)


E = -log( sig(x) ) - log( sig(-x) )

이걸 미분하면

E' = -( 1 - sig(x) ) + sig(x) = -1 + 2*sig(x)

( exp(x) - 1 ) / ( exp(x) + 1 ) 
    = exp(x) / ( exp(x) + 1 ) - ( 1 / ( exp(x) + 1 ) )
    = sig(x) - ( 1 - sig(x) ) = -1 + 2*sig(x)

• word2vec 관련 이론 정리
• Unifying Word Embeddings and Matrix Factorization — Part 1
• word2vec implementation using tensorflow
• hierarchical softmax
• original google word2vec in C
• gensim word2vec in python
  • optimization
  • parallelizing
• word2vec in tensorflow
• word2vec in c++
• word2vec in java
  • parallelizing
• word2vec using nvidia GPU
  • word2vec_cbow
    • word2vec using nvidia GPU
  • cuda-word2vec
• paragraph vector
  • paragraph in c
  • doc2vec
  • sentence2vec

2. Gensim

• how to install
  • install python2.7 (for example)

  wget --no-check-certificate http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tar.bz2
  tar xf Python-2.7.3.tar.bz2
  cd Python-2.7.3
  ./configure --prefix=/usr/local
  make && make altinstall
  wget --no-check-certificate http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg#md5=fe1f997bc722265116870bc7919059ea
  chmod +x setuptools-0.6c11-py2.7.egg
  ./setuptools-0.6c11-py2.7.egg --prefix=/usr/local
  wget --no-check-certificate https://pypi.python.org/packages/source/p/pip/pip-1.3.1.tar.gz
  tar xfz pip-1.3.1.tar.gz
  cd pip-1.3.1
  python2.7 setup.py install    
  

  • [install blas,atlas,lapack] (https://github.com/dsindex/blog/wiki/%5Btheano%5D--Deep-Learning,-Jumpto-with-Theano)
  • install numpy,scipy

  yum install numpy
  pip install numpy
  pip install scipy
  

  • install gensim
    • recent version included word2vec,doc2vec
  • install cython

  pip instal cython
  

• test sources
  • document_to_corpus.py
  • transform.py
  • similarity.py
  • dist_lsi.py
  • dist_lda.py
  • build_word2vec.py
  • search_word2vec.py
• distributed word2vec
  • word2vec in spark mllib

3. Related Topics

• from word2vec to doc2vec
• distributional semantics and word2vec
• From Word Embeddings To Document Distances
  • WMD implementation
  • FLANN