Get different results from the same image

[rubai1597]

When I used the code below, I got different results for the same image as the attached images.

import requests

from PIL import Image

import gem
import torch


@torch.no_grad()
def main() -> None:
    model_name = "ViT-L/14-336"
    pretrained = "openai"
    device = "cuda" if torch.cuda.is_available() else "cpu"

    model = gem.create_gem_model(
        model_name=model_name, pretrained=pretrained, device=device
    )
    preprocess = gem.get_gem_img_transform()

    texts = [
        "a photo of cat",
    ]
    url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    while True:
        images_pil = Image.open(requests.get(url, stream=True).raw)
        img_tensor = preprocess(images_pil).unsqueeze(0).to(device)

        logits = model.forward(img_tensor, texts)
        gem.visualize(img_tensor[0], texts, logits[0])


if __name__ == "__main__":
    main()

[rubai1597]

Here is the package list of my environment.

gem-torch                1.0.1
huggingface-hub          0.22.2
numpy                    1.26.4
nvidia-cublas-cu12       12.1.3.1
nvidia-cuda-cupti-cu12   12.1.105
nvidia-cuda-nvrtc-cu12   12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12        8.9.2.26
nvidia-cufft-cu12        11.0.2.54
nvidia-curand-cu12       10.3.2.106
nvidia-cusolver-cu12     11.4.5.107
nvidia-cusparse-cu12     12.1.0.106
nvidia-nccl-cu12         2.19.3
nvidia-nvjitlink-cu12    12.4.127
nvidia-nvtx-cu12         12.1.105
open-clip-torch          2.24.0
opencv-python            4.9.0.80
pillow                   10.3.0
torch                    2.2.2
torchvision              0.17.2
tqdm                     4.66.2

[Leminhbinh0209]

Hi, I experienced a similar issue:

I tried with the author's provided Colab Notebook and on my GPU (A100) torch: 2.0.0+cu117 | open_clip: 2.26.1, both didn't give me the expected results.
Later, I tried with other settings (3090Ti)torch: 2.0.1+cu117 | open_clip: 2.24.0 , and it works normally.

Not sure what is the reason behind. Any comments are welcome!

[WalBouss]

Hi, Thanks for the feedback!
I haven't experienced similar issues, I'll try to replicate your results and get back to you.
One guess, though, is that the open_clip library may have introduced some changes that broke compatibility with the gem's wrapper.
I'll look into that!