Falcon-40B-Instruct is a large language model with 40 billion parameters, developed by TII. It is a causal decoder-only model fine-tuned on a mixture of Baize data and is released under the Apache 2.0 license. This model is optimized for inference and features FlashAttention and multiquery architectures. It is primarily designed for chat and instruct applications in English and French. However, it may not be suitable for further fine-tuning. It is available under the Apache 2.0 license.

Model Type: Causal decoder-only Languages: English and French License: Apache 2.0 Training Data: Fine-tuned on 150 million tokens from Bai ze mixed with 5% of RefinedWeb data Architecture: Based on GPT-3 with optimizations including rotary positional embeddings, FlashAttention, and multiquery attention Hardware: Trained on AWS SageMaker using 64 A100 40GB GPUs in P4d instances Software: Utilizes a custom distributed training codebase called Gigatron

Falcon-40B-Instruct may carry biases commonly found online due to its training data. Users are advised to implement guardrails and take precautions for production use. It's mostly suited for English and French and may not generalize well to other languages.

The above summary was generated using ChatGPT. Review the original model card to understand the data used to train the model, evaluation metrics, license, intended uses, limitations and bias before using the model.

Falcon-40B-Instruct was finetuned on a 150M tokens from Bai ze mixed with 5% of RefinedWeb data.

The data was tokenized with the Falcon-7B/40B tokenizer.

Falcon-40B-Instruct was trained on AWS SageMaker, on 64 A100 40GB GPUs in P4d instances.

Paper coming soon.

See the OpenLLM Leaderboard for early results.

Falcon-40B is a causal decoder-only model trained on a causal language modeling task (i.e., predict the next token).

The architecture is broadly adapted from the GPT-3 paper (Brown et al., 2020), with the following differences:

• Positionnal embeddings: rotary (Su et al., 2021);
• Attention: multiquery (Shazeer et al., 2019) and FlashAttention (Dao et al., 2022);
• Decoder-block: parallel attention/MLP with a two layer norms.

For multiquery, we are using an internal variant which uses independent key and values per tensor parallel degree.

Falcon-40B-Instruct was trained on AWS SageMaker, on 64 A100 40GB GPUs in P4d instances.

Falcon-40B-Instruct was trained a custom distributed training codebase, Gigatron. It uses a 3D parallelism approach combined with ZeRO and high-performance Triton kernels (FlashAttention, etc.)

Falcon-40B is made available under the Apache 2.0 license.

{
  "input_data": {
      "input_string":["Develop a Python function to sort a list of integers in ascending order"]
  }
}

[
  {
    "0": "You can use the sorted() function in Python to sort a list of integers in ascending order. Here's an example: my_list = [3,1,6,4,1,5] sorted_list = sorted(my_list) print(sorted_list) This will output: [1,1,3,4,5,6]"
  }
]

Version: 5

Featured license : apache-2.0 SharedComputeCapacityEnabled task : text-generation author : tiiuae huggingface_model_id : tiiuae/falcon-40b-instruct evaluation_compute_allow_list : ['Standard_NC24s_v3', 'Standard_NC24rs_v3', 'Standard_ND40rs_v2', 'Standard_ND96asr_v4', 'Standard_ND96amsr_A100_v4'] inference_compute_allow_list : ['Standard_ND40rs_v2', 'Standard_ND96asr_v4', 'Standard_ND96amsr_A100_v4'] inference_supported_envs : ['vllm']

View in Studio: https://ml.azure.com/registries/azureml/models/tiiuae-falcon-40b-instruct/version/5

License: apache-2.0

SharedComputeCapacityEnabled: True

SHA: ca78eac0ed45bf64445ff0687fabba1598daebf3

datasets: tiiuae/falcon-refinedweb

languages: en

evaluation-min-sku-spec: 24|4|448|2900

evaluation-recommended-sku: Standard_NC24s_v3, Standard_NC24rs_v3, Standard_ND40rs_v2, Standard_ND96asr_v4, Standard_ND96amsr_A100_v4

inference-min-sku-spec: 40|8|672|2900

inference-recommended-sku: Standard_ND40rs_v2, Standard_ND96asr_v4, Standard_ND96amsr_A100_v4