Skip to content

Commit

Permalink
Merge branch 'main' of https://github.com/fedshyvana/CONCH into main
Browse files Browse the repository at this point in the history
  • Loading branch information
@fedshyvana
fedshyvana committed Aug 9, 2024
2 parents 24ce6e7 + f7bed1c commit 02d6ac5
Show file tree
Hide file tree
Showing 2 changed files with 35 additions and 21 deletions.
54 changes: 34 additions & 20 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,17 +25,24 @@ pip install --upgrade pip
pip install -e .
```

## Updates

- **07/16/2024**: Included comparisons with [Virchow](https://github.com/mahmoodlab/UNI?tab=readme-ov-file#slide-benchmarks).
- **06/15/2024**: Included comparisons with [Prov-GigaPath](https://github.com/mahmoodlab/UNI?tab=readme-ov-file#slide-benchmarks).

## Preparing and loading the model
1. Request access to the model weights from the Huggingface model page [here](https://huggingface.co/MahmoodLab/conch).

2. Download the model weights

First create the `checkpoints` directory inside the root of the repo:
```shell
mkdir -p checkpoints/conch/
```
Then download the pretrained model (`pytorch_model.bin`) and place it in the `CONCH/checkpoints/conch/` directory.

3. Loading the model

First import the model builder:
```python
from conch.open_clip_custom import create_model_from_pretrained
Expand Down Expand Up @@ -112,29 +119,33 @@ Please refer to the UNI [[1]](https://www.nature.com/articles/s41591-024-02857-3
### Slide Benchmarks
| Model name | Pretraining | EBRAINS-C (12 classes, Public) | EBRAINS-F (30 classes, Public) | PANDA (5 classes, Public) | OncoTree-108 (108 classes, Internal) | IHC ER / PR Assess. (6 classes, Internal) |
|:---------------|:------------------|---------------------------:|-------------------------:|-----------------:|------------------:|---------------------------:|
| | | Balanced acc. | Balanced acc. |Quadratic-weight $\kappa$ | Balanced acc. | Quadratic-weight $\kappa$ |
| **UNI** [[1]](https://www.nature.com/articles/s41591-024-02857-3) | Vision | **0.883** | <ins>0.675</ins> | <ins>0.946</ins> | **0.538** | 0.785 |
| **CONCH** [[2]](https://www.nature.com/articles/s41591-024-02856-4) | Vision-language | <ins>0.868</ins> | **0.689** | 0.934 | <ins>0.515</ins> | **0.819** |
| Phikon [[3]](https://doi.org/10.1101/2023.07.21.23292757) | Vision | 0.810 | 0.659 | **0.950** | 0.486 | 0.744 |
| REMEDIS [[4]](https://doi.org/10.1038/s41551-023-01049-7) | Vision | 0.687 | 0.382 | 0.932 | 0.412 | 0.762 |
| CTransPath [[5]](https://doi.org/10.1016/j.media.2022.102559) | Vision | 0.666 | 0.514 | 0.927 | 0.399 | <ins>0.786<ins> |
| Quilt-Net [[6]](https://proceedings.neurips.cc/paper_files/paper/2023/file/775ec578876fa6812c062644964b9870-Paper-Datasets_and_Benchmarks.pdf) | Vision-language | 0.728 | 0.608 | 0.909 | 0.389 | 0.784 |
| PLIP [[7]](https://doi.org/10.1038/s41591-023-02504-3) | Vision-language | 0.683 | 0.562 | 0.901 | 0.369 | 0.759 |
| ResNet-50 (Tr) [[8]](https://openaccess.thecvf.com/content_cvpr_2016/papers/He_Deep_Residual_Learning_CVPR_2016_paper.pdf) | ImageNet Transfer | 0.302 | 0.219 | 0.831 | 0.148 | 0.709 |

| | | Balanced acc. | Balanced acc. | Quadratic-weight $\kappa$ | Balanced acc. | Quadratic-weight $\kappa$ |
| **UNI** [[1]](https://www.nature.com/articles/s41591-024-02857-3) | Vision | **0.883** | 0.675 | <ins>0.946</ins> | **0.538** | 0.785 |
| **CONCH** [[2]](https://www.nature.com/articles/s41591-024-02856-4) | Vision-language | 0.868 | **0.689** | 0.934 | 0.515 | <ins>0.819</ins> |
| Virchow (CLS+MEAN) [[3]](https://arxiv.org/pdf/2309.07778) | Vision | 0.833 | 0.654 | 0.943 | 0.519 | 0.788 |
| Prov-GigaPath [[4]](https://www.nature.com/articles/s41586-024-07441-w) | Vision | <ins>0.875</ins> | <ins>0.687</ins> | 0.942 | <ins>0.522</ins> | **0.821** |
| Phikon [[5]](https://doi.org/10.1101/2023.07.21.23292757) | Vision | 0.810 | 0.659 | **0.950** | 0.486 | 0.744 |
| REMEDIS [[6]](https://doi.org/10.1038/s41551-023-01049-7) | Vision | 0.687 | 0.382 | 0.932 | 0.412 | 0.762 |
| CTransPath [[7]](https://doi.org/10.1016/j.media.2022.102559) | Vision | 0.666 | 0.514 | 0.927 | 0.399 | 0.786 |
| Quilt-Net [[8]](https://proceedings.neurips.cc/paper_files/paper/2023/file/775ec578876fa6812c062644964b9870-Paper-Datasets_and_Benchmarks.pdf) | Vision-language | 0.728 | 0.608 | 0.909 | 0.389 | 0.784 |
| PLIP [[9]](https://doi.org/10.1038/s41591-023-02504-3) | Vision-language | 0.683 | 0.562 | 0.901 | 0.369 | 0.759 |
| ResNet-50 (Tr) [[10]](https://openaccess.thecvf.com/content_cvpr_2016/papers/He_Deep_Residual_Learning_CVPR_2016_paper.pdf) | ImageNet Transfer | 0.302 | 0.219 | 0.831 | 0.148 | 0.709 |

### ROI Benchmarks
| Model name | Pretraining | CRC-100K-Raw (9 classes, Public) | TCGA Uniform Tumor (32 classes, Public) |
|:---------------|:------------------|---------------------------:|-------------------------:|
| | | Balanced acc. | Balanced acc. |
| **UNI** [[1]](https://www.nature.com/articles/s41591-024-02857-3) | Vision | <ins>0.925</ins> | **0.595** |
| **CONCH** [[2]](https://www.nature.com/articles/s41591-024-02856-4) | Vision-language | **0.941** | <ins>0.556</ins> |
| Phikon [[3]](https://doi.org/10.1101/2023.07.21.23292757) | Vision | 0.845 | 0.533 |
| REMEDIS [[4]](https://doi.org/10.1038/s41551-023-01049-7) | Vision | 0.908 | 0.541 |
| CTransPath [[5]](https://doi.org/10.1016/j.media.2022.102559) | Vision | 0.836 | 0.463 |
| Quilt-Net [[6]](https://proceedings.neurips.cc/paper_files/paper/2023/file/775ec578876fa6812c062644964b9870-Paper-Datasets_and_Benchmarks.pdf) | Vision-language | 0.878 | 0.359 |
| PLIP [[7]](https://doi.org/10.1038/s41591-023-02504-3) | Vision-language | 0.840 | 0.370 |
| ResNet-50 [[8]](https://openaccess.thecvf.com/content_cvpr_2016/papers/He_Deep_Residual_Learning_CVPR_2016_paper.pdf) | ImageNet Transfer | 0.797 | 0.318 |
| **UNI** [[1]](https://www.nature.com/articles/s41591-024-02857-3) | Vision | 0.925 | **0.595** |
| **CONCH** [[2]](https://www.nature.com/articles/s41591-024-02856-4) | Vision-language | **0.941** | 0.556 |
| Virchow (CLS+MEAN) [[3]](https://arxiv.org/pdf/2309.07778) | Vision | 0.919 | 0.549 |
| Virchow (CLS) [[3]](https://arxiv.org/pdf/2309.07778) | Vision | 0.895 | 0.544 |
| Prov-GigaPath [[4]](https://www.nature.com/articles/s41586-024-07441-w) | Vision | <ins>0.929</ins> | <ins>0.593</ins> |
| Phikon [[5]](https://doi.org/10.1101/2023.07.21.23292757) | Vision | 0.845 | 0.533 |
| REMEDIS [[6]](https://doi.org/10.1038/s41551-023-01049-7) | Vision | 0.908 | 0.541 |
| CTransPath [[7]](https://doi.org/10.1016/j.media.2022.102559) | Vision | 0.836 | 0.463 |
| Quilt-Net [[8]](https://proceedings.neurips.cc/paper_files/paper/2023/file/775ec578876fa6812c062644964b9870-Paper-Datasets_and_Benchmarks.pdf) | Vision-language | 0.878 | 0.359 |
| PLIP [[9]](https://doi.org/10.1038/s41591-023-02504-3) | Vision-language | 0.840 | 0.370 |
| ResNet-50 [[10]](https://openaccess.thecvf.com/content_cvpr_2016/papers/He_Deep_Residual_Learning_CVPR_2016_paper.pdf) | ImageNet Transfer | 0.797 | 0.318 |

## Acknowledgements

Expand All @@ -152,9 +163,12 @@ Lu, M. Y., Chen, B., Williamson, D. F., Chen, R. J., Liang, I., Ding, T., ... &

```
@article{lu2024avisionlanguage,
title={A Vision Language Foundation Model for Computational Pathology},
author={Lu, Ming Y and Chen, Bowen and Williamson, Drew FK and Chen, Richard J and Liang, Ivy and Ding, Tong and Jaume, Guillaume and Odintsov, Igor and Zhang, Andrew and Le, Long Phi and others},
title={A visual-language foundation model for computational pathology},
author={Lu, Ming Y and Chen, Bowen and Williamson, Drew FK and Chen, Richard J and Liang, Ivy and Ding, Tong and Jaume, Guillaume and Odintsov, Igor and Le, Long Phi and Gerber, Georg and others},
journal={Nature Medicine},
pages={863–874},
volume={30},
year={2024},
publisher={Nature Publishing Group}
}
```
Expand Down
2 changes: 1 addition & 1 deletion notebooks/zeroshot_retrieval_example.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Load tokenizer and specify some prompts. Simplicity we just use one prompt per class (lung adenocarcinoma vs. lung squamous cell carcinoma) here instead ensembling multiple prompts / prompt templates."
"Load tokenizer and specify some prompts."
]
},
{
Expand Down

0 comments on commit 02d6ac5

Please sign in to comment.