Digital pathology poses unique computational challenges, as a standard gigapixel slide may comprise tens of thousands of image tiles^1,^2,^3. Previous models often rely predominantly on tile-level predictions, which can overlook critical slide-level context and spatial dependencies^4. Here we present Prov-GigaPath, a whole-slide pathology foundation model pretrained on 1.3 billion 256 × 256 pathology image tiles in 171,189 whole slides from Providence, a large U.S. health network comprising 28 cancer centers.

To pretrain Prov-GigaPath, we propose GigaPath, a novel vision transformer architecture for pretraining gigapixel pathology slides. To scale GigaPath for slide-level learning with tens of thousands of image tiles, GigaPath adapts the newly developed LongNet[^5] method to digital pathology.

For additional details, please see the publication: A whole-slide foundation model for digital pathology from real-world data

For documentation and example Jupyter Notebooks, visit: Prov-GigaPath - GitHub

Prov-GigaPath processes an entire histopathology slide by analyzing individual tiles and generating semantically meaningful embedding. These embeddings can be used as features for a wide range of clinical applications. Prov-GigaPath excels in long-context modelling of gigapixel pathology slides, by distilling varied local pathological structures and integrating global signatures across the whole slide. Prov-GigaPath consists of a tile encoder for capturing local features and a slide encoder for capturing global features. The tile encoder individually projects all tiles into compact embeddings. The slide encoder then inputs the sequence of tile embeddings and generates contextualized embeddings taking into account the entire sequence using a transformer. The tile encoder is pretrained using DINOv2, the state-of-the-art image self-supervised learning framework. The slide encoder combines masked autoencoder pretraining with LongNet5, our recently developed method for ultra long-sequence modelling. In downstream tasks, the output of the slide encoder is aggregated using a simple softmax attention layer.

The License for Prov-GigaPath is a research-use-only license: prov-gigapath/LICENSE.

The model is not intended or made available for clinical use as a medical device, clinical support, diagnostic tool, or other technology intended to be used in the diagnosis, cure, mitigation, treatment, or prevention of disease or other conditions. The model is not designed or intended to be a substitute for professional medical advice, diagnosis, treatment, or judgment and should not be used as such. All users are responsible for reviewing the output of the developed model to determine whether the model meets the user’s needs and for validating and evaluating the model before any clinical use.

For questions or comments, please contact: [email protected]

To evaluate Prov-GigaPath, we construct a digital pathology benchmark comprising 9 cancer subtyping tasks and 17 pathomics tasks, using both Providence and TCGA data. With large-scale pretraining and ultra-large-context modelling, Prov-GigaPath attains state-of-the-art performance on 25 out of 26 tasks, with significant improvement over the second-best method on 18 tasks.

Table comparing Prov-GigaPath with state-of-the-art pathology foundation models on 26 tasks in pathomics and cancer subtyping using AUROC. * indicates the significance level that Prov-GigaPath outperforms the best comparison approach on the specific task, with Wilcoxon test p-value< 5 x 10-2 for *, p-value<1x10-2 for ** , p-value< 1x 10-3 for ***. The last column shows the p-value using the one-sided Wilcoxon test.

The paper showcases a handful of supplementary figures which highlight the demographic statistics of the training population.

Table: Sex distribution of patients in Prov-Path.

Table: Age distribution of patients in Prov-Path.

Table: Self-reported ethnicity distribution of patients in Prov-Path.

Responsible AI is a shared responsibility and we have identified six principles and practices help organizations address risks, innovate, and create value: fairness, reliability and safety, privacy and security, inclusiveness, transparency, and accountability. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant use case and addresses unforeseen product misuse. While testing the model with images and/or text, ensure the the data is PHI free and that there is no patient information or information that can be tracked to a patient identity.

The data, code, and model checkpoints are intended to be used solely for (I) future research on pathology foundation models and (II) reproducibility of the experimental results reported in the reference paper. The data, code, and model checkpoints are not intended to be used in clinical care or for any clinical decision-making purposes.

The primary intended use is to support AI researchers reproducing and building on top of this work. GigaPath should be helpful for exploring pre-training, and encoding of digital pathology slides data.

• Any deployed use case of the model — commercial or otherwise — is out of scope. Although we evaluated the models using a broad set of publicly-available research benchmarks, the models and evaluations are intended for research use only and not intended for deployed or clinical use cases.
• Use by clinicians to inform clinical decision-making, as a diagnostic tool, or as a medical device — GigaPath is not designed or intended to be deployed as-is in clinical settings nor is it for use in the diagnosis, cure, mitigation, treatment, or prevention of disease or other conditions (including to support clinical decision-making), or as a substitute of professional medical advice, diagnosis, treatment, or clinical judgment of a healthcare professional.
• Scenarios without consent for data — Any scenario that uses health data for a purpose for which consent was not obtained.
• Use outside of health scenarios — Any scenario that uses non-medical related image and/or serving purposes outside of the healthcare domain.

Please see Microsoft's Responsible AI Principles and approach available at Microsoft's Responsible AI Principles.

data = {
  "input_data": {
    "columns": ["image"],
    "index": list(range(len(image_paths))),
    "data": [
        [
            base64.encodebytes(read_image(path)).decode("utf-8")
        ] for path in image_paths
    ]
  }
}

Output: Outputs from the API are image embeddings users can do various downstream tasks, including PCA analysis (example below).

A de-identified sample subset of the Prov-Path data can be accessed from these links [^5],[^6].

Sample notebooks can be accessed below as well. They assume a HuggingFace distribution of the model:

• Semantic visualizations of the GigaPath tile embeddings
• Calculating GigaPath slide-level embeddings
• Fine-tuning GigaPath for downstream tasks

• Supports: CPU and GPU
• Default: Single V100 GPU or Intel CPU
• Minimum: Single GPU instance with 8GB memory (fastest) or CPU instance

[^5]https://zenodo.org/records/10909616 [^6]: https://zenodo.org/records/10909922

For more information on responsible AI practices, refer to Microsoft's Responsible AI Principles at https://www.microsoft.com/en-us/ai/principles-and-approach/.

Version: 1

task : image-feature-extraction industry : health-and-life-sciences Preview inference_supported_envs : ['hf'] license : mit author : Microsoft hiddenlayerscanned SharedComputeCapacityEnabled inference_compute_allow_list : ['Standard_NC6s_v3', 'Standard_NC12s_v3', 'Standard_NC24s_v3', 'Standard_NC24ads_A100_v4', 'Standard_NC48ads_A100_v4', 'Standard_NC96ads_A100_v4', 'Standard_ND96asr_v4', 'Standard_ND96amsr_A100_v4', 'Standard_ND40rs_v2']

View in Studio: https://ml.azure.com/registries/azureml/models/Prov-GigaPath/version/1

License: mit

inference-min-sku-spec: 6|1|112|64

inference-recommended-sku: Standard_NC6s_v3, Standard_NC12s_v3, Standard_NC24s_v3, Standard_NC24ads_A100_v4, Standard_NC48ads_A100_v4, Standard_NC96ads_A100_v4, Standard_ND96asr_v4, Standard_ND96amsr_A100_v4, Standard_ND40rs_v2

languages: en

SharedComputeCapacityEnabled: True