Private Attribute Inference from Images with Vision-Language Models

This is the accompanying code repository to the paper "Private Attribute Inference from Images with Vision-Language Models" published at NeurIPS 2024. For any questions, please feel free to contact the authors.

Folder Structure

At the top level we have:

.
├── backend
├── Experimentation
├── frontend
└── README.md

Labelling

In order to run the labelling platform, follow the README in ./backend and in ./frontend. We use a fastapi backend and a nextjs frontend. Both of them should be running and the labeller should visit: http://localhost:4000 to start labelling.

1. First go to ./backend and follow the README to install the environment and start the server.
2. Go to ./frontend and follow the README to install the environment and start the frontend server.
3. Then go to http://localhost:4000 and compile a dataset through labelling.

Once you have the dataset, follow instruction in Experimetation/dataset/README to get make the dataset ready for experiments.

Experiments

1. You can simply put the images and the dataset into the ./Experimentation/dataset folder.
2. Then you can set up your environements. There are different environments for different models.
3. Once you have the environments, you can start running the models.
4. Running models will store intermediate data which will be used to run comparison scripts.
5. Then to get the performance, you can run the performance scripts.

More details are to be found in the corresponding README.md

---

Authors:
Batuhan Tömekçe, [email protected]
Mark Vero, [email protected]
Robin Staab, [email protected]
Martin Vechev, [email protected]

License:
This project is licensed under the MIT License - see the LICENSE file for details. Note that this applies only to our code in this repository and not to any dependencies nor the data for which their respective licenses apply.

Citation:

@inproceedings{tomekce2024private,
  title={Private Attribute Inference from Images with Vision-Language Models},
  author={Tömekçe, Batuhan and Vero, Mark and Staab, Robin and Vechev, Martin},
  booktitle={Advances in Neural Information Processing Systems},
  year={2024}
}