Modern Generative AI with ChatGPT - doraithodla/notes GitHub Wiki

Valentina Alto Published by Packt Publishing GPT

Tokens - word fragments Prompts - guides the AI LM to generate response or output. The prompt can be a question, a statement, or a sentence and it is issued to provide context and direction to the LM Context - Model confidence - indicate how confident the AI model is in the correctness or relevance of its generated response to a given input prompt.

Codex - A set of models that can understand and generate code in various programming languages. Codex can translate natural language prompts into working code, making it a powerful tool for software development. Temperature - controls the randomness of model's response. max length stop sequences top probabilities embeddings

RNNs - Exploding/Vanishing gradients, No context, not parallelizable LSTM - Limited short term memory, lack of parallelization Transformers - replace recurrence with self attention mechanism allowing parallel computation.

Transformer Architecture Encoders - layers that transform natural laguage into numerical vectors using embeddings

Common Crawl (https://commoncrawl.org/): A massive corpus of web data gathered over an 8-year period with minimal filtering WebText2 (https://openwebtext2.readthedocs.io/en/latest/background/): A collection of text from web pages linked to in Reddit posts with at least 3 upvotes Books1 and Books2: Two separate corpora consisting of books available on the internet Wikipedia: A corpus containing articles from the English-language version of the popular online encyclopedia, Wikipedia Here you can get a better idea:

Figure 2.23 – GPT-3 knowledge base REferences: Paper: Improving Language Understanding by Generative Pre-Training,