Introduction - RamNayakTech/knowledge-hub GitHub Wiki

Introduction to Attention and Language Models

---
title: Evolution
---
  flowchart LR
    A["2001 Neural Language Model"] --> B["2013 - word2vec embedding"] --> C["Sequence to sequence model"]  --> D["2015 - Attention"] --> E["2017 - Transformer"] --> F["2018 - Pretrained models"]

RNN - Recurrent Neural Network - handle sequential data where order of information is crucial. Ex. Time series data. RNNs have an internal memory that allows them to remember previous inputs, which is essential for predicting the next in a sentence. FNN - Feedforward Neural Network

Bias - Direct and Indirect Bias

Long short term memory

Sequence to sequence models An encoder reads a variable length input sequence, and a decoder produces a variable length sequence output.

Self-attention

How Transformers Use Attentions to Process Text

Transfer Learning

Natural Language Understanding

Pre-training and Fine-tuning BERT

Hands-on BERT