Skip to content

Commit

Permalink
Polish benchmarks page (#1432)
Browse files Browse the repository at this point in the history
  • Loading branch information
@philippmwirth
philippmwirth authored Nov 22, 2023
1 parent 6a9d625 commit caf72c5
Show file tree
Hide file tree
Showing 2 changed files with 167 additions and 202 deletions.
82 changes: 25 additions & 57 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -262,23 +262,23 @@ We provide multi-GPU training examples with distributed gather and synchronized
## Benchmarks

Implemented models and their performance on various datasets. Hyperparameters are not
tuned for maximum accuracy. For detailed results and more info about the benchmarks click
tuned for maximum accuracy. For detailed results and more information about the benchmarks click
[here](https://docs.lightly.ai/self-supervised-learning/getting_started/benchmarks.html).


### Imagenet
### ImageNet1k

The following experiments have been conducted on a system with 2x4090 GPUs.
Training a model takes around 4 days for 100 epochs (35 min per epoch), including kNN, linear probing, and fine-tuning evaluation.
[ImageNet1k benchmarks](https://docs.lightly.ai/self-supervised-learning/getting_started/benchmarks.html#imagenet1k)

> **Note**: Evaluation settings are based on these papers:
> * Linear: [SimCLR](https://arxiv.org/abs/2002.05709)
> * Finetune: [SimCLR](https://arxiv.org/abs/2002.05709)
> * KNN: [InstDisc](https://arxiv.org/abs/1805.01978)
>
> See the [benchmarking scripts](./benchmarks/imagenet/resnet50/) for details.
**Note**: Evaluation settings are based on these papers:
* Linear: [SimCLR](https://arxiv.org/abs/2002.05709)
* Finetune: [SimCLR](https://arxiv.org/abs/2002.05709)
* KNN: [InstDisc](https://arxiv.org/abs/1805.01978)

See the [benchmarking scripts](./benchmarks/imagenet/resnet50/) for details.

| Model | Backbone | Batch Size | Epochs | Linear Top1 | Finetune Top1 | KNN Top1 | Tensorboard | Checkpoint |

| Model | Backbone | Batch Size | Epochs | Linear Top1 | Finetune Top1 | kNN Top1 | Tensorboard | Checkpoint |
|----------------|----------|------------|--------|-------------|---------------|----------|-------------|------------|
| BarlowTwins | Res50 | 256 | 100 | 62.9 | 72.6 | 45.6 | [link](https://tensorboard.dev/experiment/NxyNRiQsQjWZ82I9b0PvKg/) | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_barlowtwins_2023-08-18_00-11-03/pretrain/version_0/checkpoints/epoch%3D99-step%3D500400.ckpt) |
| BYOL | Res50 | 256 | 100 | 62.4 | 74.0 | 45.6 | [link](https://tensorboard.dev/experiment/Z0iG2JLaTJe5nuBD7DK1bg) | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_byol_2023-07-10_10-37-32/pretrain/version_0/checkpoints/epoch%3D99-step%3D500400.ckpt) |
Expand All @@ -290,52 +290,20 @@ Training a model takes around 4 days for 100 epochs (35 min per epoch), includin
| VICReg | Res50 | 256 | 100 | 63.0 | 73.7 | 46.3 | [link](https://tensorboard.dev/experiment/qH5uywJbTJSzgCEfxc7yUw) | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_vicreg_2023-09-11_10-53-08/pretrain/version_0/checkpoints/epoch%3D99-step%3D500400.ckpt) |

*\*We use square root learning rate scaling instead of linear scaling as it yields
better results for smaller batch sizes. See Appendix B.1 in [SimCLR paper](https://arxiv.org/abs/2002.05709).*



### ImageNette

| Model | Backbone | Batch Size | Epochs | KNN Top1 |
|-------------|----------|------------|--------|----------|
| BarlowTwins | Res18 | 256 | 800 | 0.852 |
| BYOL | Res18 | 256 | 800 | 0.887 |
| DCL | Res18 | 256 | 800 | 0.861 |
| DCLW | Res18 | 256 | 800 | 0.865 |
| DINO | Res18 | 256 | 800 | 0.888 |
| FastSiam | Res18 | 256 | 800 | 0.873 |
| MAE | ViT-S | 256 | 800 | 0.610 |
| MSN | ViT-S | 256 | 800 | 0.828 |
| Moco | Res18 | 256 | 800 | 0.874 |
| NNCLR | Res18 | 256 | 800 | 0.884 |
| PMSN | ViT-S | 256 | 800 | 0.822 |
| SimCLR | Res18 | 256 | 800 | 0.889 |
| SimMIM | ViT-B32 | 256 | 800 | 0.343 |
| SimSiam | Res18 | 256 | 800 | 0.872 |
| SwaV | Res18 | 256 | 800 | 0.902 |
| SwaVQueue | Res18 | 256 | 800 | 0.890 |
| SMoG | Res18 | 256 | 800 | 0.788 |
| TiCo | Res18 | 256 | 800 | 0.856 |
| VICReg | Res18 | 256 | 800 | 0.845 |
| VICRegL | Res18 | 256 | 800 | 0.778 |


### Cifar10

| Model | Backbone | Batch Size | Epochs | KNN Top1 |
|-------------|----------|------------|--------|----------|
| BarlowTwins | Res18 | 512 | 800 | 0.859 |
| BYOL | Res18 | 512 | 800 | 0.910 |
| DCL | Res18 | 512 | 800 | 0.874 |
| DCLW | Res18 | 512 | 800 | 0.871 |
| DINO | Res18 | 512 | 800 | 0.848 |
| FastSiam | Res18 | 512 | 800 | 0.902 |
| Moco | Res18 | 512 | 800 | 0.899 |
| NNCLR | Res18 | 512 | 800 | 0.892 |
| SimCLR | Res18 | 512 | 800 | 0.879 |
| SimSiam | Res18 | 512 | 800 | 0.904 |
| SwaV | Res18 | 512 | 800 | 0.884 |
| SMoG | Res18 | 512 | 800 | 0.800 |
better results for smaller batch sizes. See Appendix B.1 in the [SimCLR paper](https://arxiv.org/abs/2002.05709).*

### ImageNet100
[ImageNet100 benchmarks](https://docs.lightly.ai/self-supervised-learning/getting_started/benchmarks.html#imagenet100)


### Imagenette

[Imagenette benchmarks](https://docs.lightly.ai/self-supervised-learning/getting_started/benchmarks.html#imagenette)


### CIFAR-10

[CIFAR-10 benchmarks](https://docs.lightly.ai/self-supervised-learning/getting_started/benchmarks.html#cifar-10)


## Terminology
Expand Down
Loading

0 comments on commit caf72c5

Please sign in to comment.