chatgpt perplexity bard - ghdrako/doc_snipets GitHub Wiki

https://platform.openai.com/tokenizer - verify number of tokens

  • 1 token ~= 4 chars in English
  • 1 token ~= ¾ words
  • 100 tokens ~= 75 words

Or

  • 1-2 sentence ~= 30 tokens
  • 1 paragraph ~= 100 tokens
  • 1,500 words ~= 2048 tokens